Deep learning feature extraction method for eVTOL field environment

Abstract

The invention provides a deep learning feature extraction method for an eVTOL field environment, and the method comprises the steps: carrying out the pre-training of a feature detection model based on synthetic data; based on an image saliency detection method, automatically marking a real field down-looking training image; training the feature extraction model by using the real field down-looking training image marked with the feature points; inputting a down-looking image actually shot by the eVTOL into the feature extraction model to obtain image feature points and feature point descriptors; according to the method, on the basis of feature detection and model pre-training of synthetic data, automatic labeling based on an image saliency detection method and feature extraction model training using difficult negative sample mining are combined, so that the training efficiency and performance of a deep learning model in an eVTOL field environment feature extraction task are integrally improved; and the actual requirements of eVTOL field mapping and positioning can be better met.

Description

Technical Field

[0001] The present invention relates to the technical field of mapping and positioning of eVTOL (electric vertical take-off and landing) field environments, and in particular to a deep learning feature extraction method for eVTOL field environments. Background Art

[0002] Traditional feature extraction methods have significant drawbacks in field mapping and positioning for eVTOL (electric Vertical Take-off and Landing) vehicles. They are susceptible to changes in lighting, viewing angle, and seasonality. Furthermore, when performing long-term positioning tasks for eVTOLs, the field conditions are highly variable, making traditional methods unable to meet these requirements and providing stable and reliable feature extraction results.

[0003] Existing deep learning feature extraction methods are mainly applicable to structured scenes such as cities and indoors. There is a lack of effective solutions to the data noise problem caused by dense forests and repeated fields in wild scenes, resulting in poor feature extraction accuracy and robustness in wild environments.

[0004] In addition, existing deep learning feature extraction methods use dense computing methods, which have low training efficiency. When faced with large-scale data training, they require a lot of computing resources and time, and cannot meet the real-time and high efficiency requirements in practical applications. Summary of the Invention

[0005] The purpose of the present invention is to provide a deep learning feature extraction method for eVTOL field environments, so as to address the deficiencies of existing technologies in feature extraction in eVTOL field environment mapping and positioning tasks, improve the robustness of feature extraction, reduce annotation costs, and improve training efficiency.

[0006] The following is a brief summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceivable aspects and is neither intended to identify key or critical elements of all aspects nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that will be provided later.

[0007] According to one aspect of the present invention, a deep learning feature extraction method for an eVTOL field environment is provided, comprising: Step A100: pre-training a feature detection model based on synthetic data; Step A200: Automatically labeling a real field downward-looking training image based on an image saliency detection method; Step A300: training a feature extraction model using a real field downward-looking training image with marked feature points; Step B400: Input the real-view image taken by the eVTOL into the feature extraction model to obtain image feature points and feature point descriptors.

[0008] In one embodiment, step A100 includes: Step A110: Generate a synthetic image using simple geometric shapes or random textures, and mark the locations of feature points; Step A120: Input the synthesized image and feature point positions into a feature detection model for pre-training.

[0009] In one embodiment, the image saliency detection method in step A200 specifically includes: Step A211: Convert the image from RGB color space to LAB color space according to the preprocessing steps of saliency detection, and calculate color saliency; Step A212: convert the image into a grayscale image, perform Gaussian blurring, and then use the Canny algorithm to extract edges to obtain an edge map; Step A213: normalize the color saliency and edge maps to the same range, and perform weighted fusion according to the set weights to obtain a comprehensive saliency map; Step A214: binarize the comprehensive saliency map according to a set threshold to obtain a preliminary saliency mask; Step A215: Calculate the connected areas of the mask through connectivity analysis, traverse the connected areas, and mark the areas with an area greater than a preset area threshold as salient areas to form a final saliency mask.

[0010] In one embodiment, the automatic labeling of the real field bottom view image in step A200 includes: Step A221: performing multiple homography transformations on a single real field downward-viewing image; Step A222: Using a pre-trained feature detection model in combination with a saliency mask, perform feature detection on multiple transformed images to obtain a feature map of each transformed image; Step A223: transform the feature map back to the original image position according to the homography matrix corresponding to the image transformation; Step A224: superimpose the feature maps transformed back to the original image position to fuse the feature information of multiple images.

[0011] Step A225: extract the coordinates of the largest K values from the superimposed feature map, and these coordinates are determined as feature point labels.

[0012] In one embodiment, the number of homography transformations in step A221 and the K value in step A225 are set according to actual needs.

[0013] In one embodiment, step A300 specifically includes: Step A310: construct an image feature encoder based on the VGG network model; Step A320: predicting the feature point positions and feature descriptors in the real field downward-looking training image with the feature points marked thereon, respectively, using a feature point decoder and a descriptor decoder; Step A330: Calculate the cross entropy loss between the label coordinates and the decoded coordinates to optimize the feature point decoder; calculate the descriptor matching loss to optimize the descriptor decoder.

[0014] In one embodiment, in step A330 , a triple-based loss function is used to calculate the descriptor matching loss.

[0015] In one embodiment, the calculation of the descriptor matching loss in step A330 further includes calculating a positive sample loss and calculating a negative sample loss based on difficult negative sample mining.

[0016] In one embodiment, calculating the negative sample loss based on difficult negative sample mining includes: Step A331: For each target data point in the training data set, calculate its feature distance with all other data points in the data set; Step A332: From all the calculated distances, select several data points with the longest distances as negative samples; Step A333: Generate a corresponding negative sample set for each data point in the data set; Step A334: Calculate the negative sample loss.

[0017] In one embodiment, the feature distance in step A331 is calculated using Euclidean distance or cosine distance.

[0018] The beneficial effects of the embodiments of the present invention are: This paper adopts a self-supervised learning image feature point labeling method based on saliency detection, avoiding the high cost and low efficiency of manual labeling. At the same time, it uses saliency detection to exclude noisy areas, improves the robustness of feature extraction, and adapts to the complex and changeable environment of eVTOL in the wild.

[0019] Through the sparse descriptor similarity loss calculation method based on difficult negative sample mining, the model's ability to distinguish similar features is enhanced, the discrimination ability of the feature space is improved, and the feature extraction results are more accurate and reliable.

[0020] Feature detection and model pre-training based on synthetic data, combined with automatic labeling based on image saliency detection methods and feature extraction model training using difficult negative sample mining, have overall improved the training efficiency and performance of deep learning models in eVTOL field environment feature extraction tasks, and can better meet the actual needs of eVTOL field mapping and positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0022] The above features and advantages of the present invention will be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or characteristics may have the same or similar reference numerals.

[0023] Figure 1 This is a flowchart of a deep learning feature extraction method for eVTOL field environments disclosed in an embodiment of the present invention; Figure 2 It is a flowchart of the overall method steps of an embodiment of the present invention; Figure 3 is a flow chart of the automatic annotation method according to an embodiment of the present invention; Figure 4 This is a flow chart of feature extraction model training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," "third," "fourth," etc. in the description and claims of the present invention are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having," as well as any variations thereof, in the embodiments of the present invention, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or are inherent to these processes, methods, products, or apparatuses.

[0026] See also Figure 1 and Figure 2 , Figure 1 This is a flowchart of a deep learning feature extraction method for eVTOL field environments disclosed in an embodiment of the present invention; Figure 2 It is a flowchart of the overall method steps of an embodiment of the present invention.

[0027] An embodiment of the present invention provides a deep learning feature extraction method for an eVTOL field environment, comprising the following steps: Step A100: pre-training a feature detection model based on synthetic data; Step A200: Automatically labeling a real field downward-looking training image based on an image saliency detection method; Step A300: training a feature extraction model using a real field downward-looking training image with marked feature points; Step B400: Input the real-view image taken by the eVTOL into the feature extraction model to obtain image feature points and feature point descriptors.

[0028] Steps A100 to A300 are offline model training, performed in a non-real-time environment, temporarily separated from the actual application scenario. Before training, a large amount of synthetic imagery and real-world field imagery data is collected. Model training is then performed on specialized computing equipment (such as a server equipped with a high-performance GPU). The entire training process is independent of real-time data collected by the eVTOL, and upon completion, a model with defined parameters is obtained.

[0029] Step B400 is online, real-time feature extraction, performed in real-time, in sync with the actual application scenario. During flight, the eVTOL utilizes the offline trained model to process the real-world imagery captured by the aircraft, instantly extracting feature points and descriptors to meet real-time positioning and mapping requirements.

[0030] Step A100 specifically involves using a synthetic image generation tool to generate synthetic images containing simple geometric shapes (such as triangles, ellipses, rectangles, etc.) or random textures. Feature point locations are annotated for these synthetic images. This annotation information is used to pre-train a feature detection model consisting of a shared encoder, a feature point detection decoder, and a descriptor generation decoder. During pre-training, network parameters are adjusted based on the synthetic image input and the annotated feature point locations, enabling the network to initially learn the capabilities of feature point detection and descriptor generation. This method requires no manual annotation and is suitable for training on large-scale data.

[0031] In step A200, a self-supervised learning image feature point annotation method based on saliency detection is employed. In traditional vision, saliency detection is typically used to extract the areas of an image that most attract human visual attention. These areas typically contain key information in the scene, such as object edges, texture variations, or areas of strong color contrast. This method proposes applying saliency detection to the mapping and localization of eVTOL outdoor scenes. It automatically identifies key feature points in the environment, excludes noisy areas that are not of interest to mapping and localization (such as leaves or repetitive mud and grass in fields), and improves the robustness of feature extraction.

[0032] In this embodiment, the saliency detection method specifically includes: Step A211: Convert the image from RGB color space to LAB color space according to the preprocessing steps of saliency detection, and calculate color saliency; Step A212: convert the image into a grayscale image, perform Gaussian blurring, and then use the Canny algorithm to extract edges to obtain an edge map; Step A213: normalize the color saliency and edge maps to the same range, and perform weighted fusion according to the set weights to obtain a comprehensive saliency map; Step A214: binarize the comprehensive saliency map according to a set threshold to obtain a preliminary saliency mask; Step A215: Calculate the connected areas of the mask through connectivity analysis, traverse the connected areas, and mark the areas with an area greater than a preset area threshold as salient areas to form a final saliency mask.

[0033] Some exemplary pseudo codes of the saliency detection method are as follows: Function: Input: I - input image (RGB) Output: M-Saliency Mask (same size as I, significant area = 1, non-significant area = 0) 1. Preprocessing: 1.1 Read the input image I and convert it to LAB color space 1.2 Calculate color saliency S_color: -Calculate the mean μ_A, μ_B of channels A and B -Calculate the color contrast of each pixel (x, y): S_color (x, y) = (A(x, y) - μ_A)^2 + (B(x, y) - μ_B)^2 - Normalize S_color to [0,255] 2. Structural saliency detection: 2.1 Convert I to grayscale image G 2.2 Perform Gaussian blur to obtain G_blur 2.3 Use the Canny algorithm to extract edges and obtain edge graph E 3. Calculate the comprehensive saliency map S_final: 3.1 Normalize S_color and E to the same range 3.2 Calculate the weighted fusion S_final = a * S_color + b * E, where a and b are weight parameters of the two, which can be adjusted according to actual conditions to balance the contribution of color and structure information to saliency.

[0034] 3.3 Binarize S_final according to the set threshold, convert it into a black and white image, and obtain the preliminary saliency mask Mask_raw 4. Connectivity Analysis: 4.1 Calculate the connected area of Mask_raw. The connected area refers to the set of interconnected pixels in the image.

[0035] 4.2 Initialize the final mask Mask = Create an all-zero matrix of the same size as the input image 4.3 Traverse all connected regions i (starting from 1, skipping the background area): If statistic[i, area] > areathreshold: Mark the current connected area as a salient area Mask[component label == i] = 1 5. Morphological optimization: 5.1 Perform dilation operation Mask = Morphological Dilation(Mask, Kernel Size = 5 × 5, Iterations = 1) 6. Return the saliency mask Mask End Function After obtaining the saliency mask, step A200 also includes automatically labeling the real field image, such as Figure 3 As shown, this step includes: Step A221: Perform multiple homography transformations on a single real field image. The homography transformation can be achieved by setting different transformation parameters (such as rotation angle, translation amount, scaling ratio, etc.). Multiple homography transformations can simulate image changes under different viewing angles. The number of homography transformations can be determined according to actual needs. Step A222: Utilize the pre-trained feature detection model, combined with a saliency mask to eliminate noise areas, and perform feature detection on multiple transformed images to obtain a feature map for each transformed image. A feature map is the result of a convolutional neural network (CNN) processing an image through operations such as convolutional layers and pooling layers. It is essentially a multidimensional array that represents the feature information of an image at different levels. Step A223: transform the feature map back to the original image position according to the homography matrix corresponding to the image transformation to ensure the correspondence between the feature map and the original image; Step A224: Overlay the feature maps transformed back to the original image positions to fuse the feature information of multiple images. After overlaying the feature maps of multiple transformed images, a new feature map is generated. This overlaid feature map integrates the feature information of multiple images, and the value at each position is the accumulation or other combination of the values at that position in the multiple feature maps.

[0036] Step A225: Extract the coordinates of the K largest values from the superimposed feature map. These coordinates are determined as feature point labels. Specifically, to identify the most representative feature points from the superimposed feature map, all elements in the feature map are sorted, and then the K elements with the largest values are selected. K here is a pre-set parameter that determines the number of feature points ultimately extracted. It is important to note that it is not the maximum values themselves that are extracted, but rather their coordinate positions in the feature map. These coordinates correspond to the locations of the most prominent feature points in the original image, and these coordinates serve as feature point labels for subsequent tasks.

[0037] For example, suppose the superimposed feature map is a 5x5 two-dimensional array as shown below: [[ 2, 5, 3, 1, 4], [ 7, 9, 6, 2, 3], [ 4, 5, 8, 3, 1], [ 1, 2, 3, 4, 5], [ 6, 7, 2, 3, 4]] If K = 3, the three largest values are 9, 8, and 7, and their coordinates in the feature map are (1, 1), (2, 2), and (1, 0) (the coordinates are represented row first and then column). These coordinates will be extracted as feature point labels.

[0038] Determining the K value requires comprehensive consideration of various factors. For eVTOL field mapping, which requires a large number of feature points to accurately reproduce the environment, a larger K value is recommended. For simple object recognition, which requires a smaller number of key feature points, a smaller K value is recommended. For example, to assist in the localization of specific landmarks in the wild, focusing on key contour features, a K value of 20-50 may be sufficient. However, for building large-scale, high-precision maps, a K value of several hundred or even thousands may be sufficient.

[0039] If the images in the dataset are complex and feature rich and diverse, such as field images containing a variety of landforms and abundant vegetation, a large K value is recommended to fully capture these features. If the images have simple features but high similarity, such as images of large, flat fields, a smaller K value can be used. For example, if the dataset consists primarily of complex mountainous images, a K value of 100-200 might be sufficient. If the dataset consists primarily of simple grassland scenes, a K value of 10-30 is sufficient.

[0040] During training, you can set an initial K value, such as 30, to train the model and evaluate performance metrics on the validation set, such as mapping and localization accuracy and object recognition accuracy. Adjust the K value based on the evaluation results. If performance improves, continue fine-tuning; if performance degrades, adjust in the opposite direction. For example, if the initial K value of 30 results in large mapping errors, increase it to 50 and retest. Repeat adjustments to find the K value that achieves optimal performance.

[0041] Figure 4 The feature extraction model structure and training process are shown, such as Figure 4 As shown, step A300 includes: Step A310: constructing an image feature encoder based on the VGG network model. The VGG network performs well in image feature extraction and can effectively extract rich feature information from the input image. Step A320: predicting the feature point positions and feature descriptors in the real field downward-looking training image with the feature points marked thereon, respectively, using a feature point decoder and a descriptor decoder; Step A330: Calculate the cross entropy loss between the label coordinates and the decoded coordinates to optimize the feature point decoder; calculate the descriptor matching loss to optimize the descriptor decoder. These losses work together to guide the model parameter update, enabling the model to more accurately extract features.

[0042] In this embodiment, a triplet-based loss function (Triplet Loss) is used to calculate the descriptor matching loss. By minimizing the loss function, the discriminative ability of the feature space is improved. A triplet consists of three samples: an anchor sample (Anchor, A ), a positive sample (Positive, P ) and a negative sample (Negative, N ). The positive sample and the anchor sample belong to the same category, and the negative sample and the anchor sample belong to different categories.

[0043] The triplet-based loss function is usually defined as: L triplet =max( d ( A , P )− d ( A , N )+ α ,0) Among them, d(A,P) is the distance between the anchor sample A and the positive sample P (such as the Euclidean distance), d(A,N) is the distance between the anchor sample A and the negative sample N, and α is a positive margin value used to ensure that there is enough space between samples of different classes.

[0044] If d(A,P)−d(A,N)+α<0, it means that the distance between the negative sample and the anchor sample is far enough, and the distance between the positive sample and the anchor sample is close enough. In this case, the loss is 0 and the model does not need to be adjusted.

[0045] If d(A,P)−d(A,N)+α≥0, it means that the distance between the negative sample and the anchor sample is not far enough, and the distance between the positive sample and the anchor sample is not close enough. At this time, the loss is greater than 0, and the model needs to adjust the parameters to reduce d(A,P) and increase d(A,N).

[0046] The calculation of the descriptor matching loss in step A330 further includes calculating the positive sample loss and calculating the negative sample loss based on difficult negative sample mining.

[0047] In order to improve the model performance and enhance its discrimination ability, this embodiment adopts a negative sample loss calculation method based on difficult negative sample mining, including: Step A331: For each target data point in the training dataset, calculate the feature distance between it and all other data points in the dataset. There are various methods for calculating feature distance, such as Euclidean distance and cosine distance, which measure the degree of feature similarity between two data points. A closer distance indicates more similar features between the two data points; a greater distance indicates greater feature dissimilarity.

[0048] Step A332: From all calculated distances, select the data points with the greatest distances as negative samples. These negative samples have significantly different characteristics from the target data points, making them easily distinguishable in traditional training. However, the key to mining hard negative samples lies in discovering samples that are similar to, but not identical to, positive samples (samples belonging to the same category as the target data point). Therefore, selecting the farthest samples as negative samples here is only a preliminary screening step, and further adjustments will be made later when constructing the negative sample set.

[0049] Step A333: Generate a corresponding negative sample set for each data point in the data set; the negative sample set formula is as follows:

[0050] This formula means that for each data point d i (in d i ∈ D , D represents the entire dataset), N i is its negative sample set, which contains d i The descriptor with the farthest distance K Descriptors ( K is a pre-set parameter that determines the number of negative samples for each data point. By constructing a negative sample set in this way, the model can pay more attention to samples that are difficult to distinguish during training, thereby enhancing the model's ability to distinguish similar features.

[0051] Step A334: Calculate the negative sample loss, incorporate it into the loss function of the entire model, and perform backpropagation together with other losses (such as the cross entropy loss between label coordinates and decoding coordinates, descriptor matching loss, positive sample loss, etc.) to update the model parameters, thereby improving the model performance.

[0052] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0053] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0054] The above description is only a preferred example of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A deep learning feature extraction method for eVTOL field environments, characterized by: include: Step A100: pre-training a feature detection model based on synthetic data; Step A200: Automatically labeling a real field downward-looking training image based on an image saliency detection method; Step A300: training a feature extraction model using a real field downward-looking training image with marked feature points; Step B400: Input the real-view image taken by the eVTOL into the feature extraction model to obtain image feature points and feature point descriptors.

2. The deep learning feature extraction method for eVTOL field environment according to claim 1 is characterized in that: The step A100 includes: Step A110: Generate a synthetic image using simple geometric shapes or random textures, and mark the locations of feature points; Step A120: Input the synthesized image and feature point positions into a feature detection model for pre-training.

3. The deep learning feature extraction method for eVTOL field environment according to claim 1, characterized in that: The image saliency detection method in step A200 specifically includes: Step A211: Convert the image from RGB color space to LAB color space according to the preprocessing steps of saliency detection, and calculate color saliency; Step A212: convert the image into a grayscale image, perform Gaussian blur, and then use the Canny algorithm to extract edges to obtain an edge map; Step A213: normalize the color saliency and edge maps to the same range, and perform weighted fusion according to the set weights to obtain a comprehensive saliency map; Step A214: binarize the comprehensive saliency map according to a set threshold to obtain a preliminary saliency mask; Step A215: Calculate the connected areas of the mask through connectivity analysis, traverse the connected areas, and mark the areas with an area greater than a preset area threshold as salient areas to form a final saliency mask.

4. The deep learning feature extraction method for eVTOL field environment according to claim 3 is characterized in that: The automatic labeling of the real field viewing image in step A200 includes: Step A221: performing multiple homography transformations on a single real field downward-viewing image; Step A222: Using a pre-trained feature detection model in combination with a saliency mask, perform feature detection on multiple transformed images to obtain a feature map of each transformed image; Step A223: transform the feature map back to the original image position according to the homography matrix corresponding to the image transformation; Step A224: superimpose the feature maps transformed back to the original image position to fuse the feature information of multiple images; Step A225: extract the coordinates of the largest K values from the superimposed feature map, and these coordinates are determined as feature point labels.

5. The deep learning feature extraction method for eVTOL field environment according to claim 4, characterized in that: The number of homography transformations in step A221 and the K value in step A225 are set according to actual needs.

6. The deep learning feature extraction method for eVTOL field environment according to claim 1, characterized in that: The step A300 specifically includes: Step A310: construct an image feature encoder based on the VGG network model; Step A320: predicting the feature point positions and feature descriptors in the real field downward-looking training image with the feature points marked thereon, respectively, using a feature point decoder and a descriptor decoder; Step A330: Calculate the cross entropy loss between the label coordinates and the decoded coordinates to optimize the feature point decoder; calculate the descriptor matching loss to optimize the descriptor decoder.

7. The deep learning feature extraction method for eVTOL field environment according to claim 6, characterized in that: In step A330 , a triple-based loss function is used to calculate the descriptor matching loss.

8. The deep learning feature extraction method for eVTOL field environment according to claim 7, characterized in that: The calculation of the descriptor matching loss in step A330 further includes calculating the positive sample loss and calculating the negative sample loss based on difficult negative sample mining.

9. The deep learning feature extraction method for eVTOL field environment according to claim 8, characterized in that: The calculation of negative sample loss based on difficult negative sample mining includes: Step A331: For each target data point in the training data set, calculate its feature distance with all other data points in the data set; Step A332: From all the calculated distances, select several data points with the longest distances as negative samples; Step A333: Generate a corresponding negative sample set for each data point in the data set; Step A334: Calculate the negative sample loss.

10. The deep learning feature extraction method for eVTOL field environment according to claim 9, characterized in that: The feature distance in step A331 is calculated using Euclidean distance or cosine distance.

Images