An unmanned aerial vehicle image unsupervised classification method integrating multi-angle information

Abstract

The application relates to an unmanned aerial vehicle image unsupervised classification method integrating multi-angle information and relates to the field of image processing.The application is used to solve the problem of low ground object classification accuracy of the existing unmanned aerial vehicle image classification method.The application comprises the following steps: segmenting multi-angle unmanned aerial vehicle images to obtain segmentation results and segmentation block corresponding semantic vectors; establishing a region approximation neighborhood module to obtain a region approximation neighborhood representative; establishing a direction link semantic knowledge graph generation module to obtain a direction link semantic knowledge graph; obtaining the knowledge graph corresponding to all segmentation blocks in the orthographic angle unmanned aerial vehicle image, so as to form a global knowledge graph list; establishing a multi-direction embedding module of the knowledge graph to obtain expression vectors consistent in structure; constructing a clustering model to cluster the expression vectors consistent in structure to obtain a clustering result; and applying the clustering model to the orthographic angle unmanned aerial vehicle image to generate an unsupervised classification result graph.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to an unsupervised classification method for UAV images that integrates multi-angle information. Background Technology

[0002] High-resolution UAV data has become an indispensable information source for critical social tasks such as resource surveys, refined urban management, natural disaster emergency response, and agricultural planting management. However, the rapid and accurate identification of land cover types within massive datasets remains a significant challenge. Unsupervised classification can accurately distinguish land covers without the need for complex labeling and sample preparation, which has significant strategic importance and practical value for improving information acquisition efficiency and supporting scientific decision-making. Therefore, conducting unsupervised classification of UAVs has important theoretical and social value.

[0003] With the widespread application of UAV remote sensing technology, unsupervised classification aims to automatically classify pixels or objects in images into different land cover categories without the need for prior manual annotation. Currently, the data perspective in the field of unsupervised image classification can be mainly categorized into two main types:

[0004] The first type of method is direct classification based on single-view orthophoto UAV imagery. This method produces a planar map with a uniform scale, giving it a significant advantage in surveying and area measurement. However, this method suffers from the problem of feature confusion due to the single viewpoint. For example, from directly above, dense tree canopies and green sports fields are highly similar in spectrum and texture; different types of buildings with similar roof materials and colors (such as factories, warehouses, and residential buildings) are also difficult to distinguish. Furthermore, orthophotos are essentially two-dimensional projections of the top surface of features, completely losing information about the vertical structure of buildings, such as facades and side walls. This makes it impossible to distinguish building height, layers, and specific outlines when classifying urban features, often incorrectly grouping high-rise buildings with their attached low-rise buildings, parking lots, etc., resulting in serious undersegmentation or misclassification. In addition, shadows caused by lighting and shooting angle completely obscure some feature information, further reducing classification accuracy.

[0005] The second type of method directly fuses multi-view information. To overcome the limitations of single-view images, researchers have introduced multi-angle images such as UAV oblique photography. The core idea of ​​traditional multi-view fusion technology is to geometrically register all oblique angle images using photogrammetry and 3D reconstruction techniques from computer vision, and then "stretch" or "project" them onto the geographic coordinate system of orthophotos to generate so-called multi-angle fused orthophotos. However, while this method theoretically integrates more information, it suffers from two fundamental flaws in practice: First, there is a loss of key perspective information. The value of oblique images lies in their unique perspective, clearly showing the facade texture of buildings, window structures, and the side outlines of vegetation. When these are forcibly projected onto a two-dimensional orthophoto plane through geometric transformation, this unique perspective information is severely distorted or even lost. The originally clear facade features are compressed into blurry lines, meaning that while existing methods introduce multi-view data, they do not solve the problem of low classification accuracy, thus losing their distinguishing significance and violating the original intention of introducing multi-view data. Second, there is the accumulation and propagation of errors. A rigorous geometric registration process involves multiple complex steps, including feature point extraction, matching, and camera parameter calculation. Each step can introduce errors, and these errors accumulate and propagate throughout the processing chain. Ultimately, this leads to ghosting, misalignment, and blurring in the fused image or model, which in turn interferes with subsequent classification tasks, resulting in even lower classification accuracy. Summary of the Invention

[0006] The purpose of this invention is to address the problem of low accuracy in ground feature classification in existing UAV image classification methods, and to propose an unsupervised UAV image classification method that integrates multi-angle information.

[0007] An unsupervised classification method for UAV imagery integrating multi-angle information, the specific process of which includes:

[0008] S1, Input multi-angle UAV images, segment each image to obtain the segmentation block corresponding to each image, thereby obtaining the segmentation result list OrthoSegList of the ortho-angle UAV images and the segmentation result list NonXSegList of the non-ortho-angle UAV images, convert each segmentation block into a feature vector with semantic information, and obtain the semantic vector list OrthoVectorList and NonXVectorList.

[0009] The multi-angle drone imagery includes: an orthoimage of a drone at an orthogonal angle and a list of non-orthoimages, NonOrthoimageList.

[0010] S2, establish the regional approximation neighborhood module RPNM. RPNM uses semantic vector InputVector, target non-orthophoto UAV image NonOrthoimageY, NonXSegList and NonXVectorList to obtain the regional myopia neighborhood representative RepVectorList;

[0011] S3, establish the direction link semantic knowledge graph generation module DESLGGM. DESLGGM uses OrthoSegList, NonOrthoimageList, OrthoVectorList and RPNM to create the direction link semantic knowledge graph KG;

[0012] S4. Use OrthoSegList, NonOrthoimageList and DESLGGM to obtain the knowledge graphs corresponding to all segments in OrthoSegList. The knowledge graphs corresponding to all segments in OrthoSegList form a global knowledge graph list GraphList.

[0013] S5, establish the knowledge graph orientation embedding module MEMKG. MEMKG uses GraphList and NonOrthoimageList to obtain the structurally consistent representation vector YZVector.

[0014] S6. Construct a clustering model KMMode. KMMode calls MEMKG to process the graph elements in GraphList, obtains structurally consistent representation vectors YZVector, associates the graph elements in GraphList with YZVector, and clusters the YZVector associated with all graph elements in GraphList to obtain the clustering results.

[0015] S7. The clustering model KMMode is applied to the orthoimage of the UAV at an orthophoto angle and the list NonOrthoimageList consisting of a set of UAV images at non-orthophoto angles to generate an unsupervised classification result image.

[0016] Further, in S1, the input multi-angle UAV images are segmented to obtain segmentation blocks corresponding to each image, thereby obtaining a segmentation result list OrthoSegList for orthogonal angle UAV images and a segmentation result list NonXSegList for non-orthogonal angle UAV images. Each segmentation block is converted into a feature vector with semantic information to obtain semantic vector lists OrthoVectorList and NonXVectorList, specifically:

[0017] S101, Input a single orthophoto drone image (Orthoimage), Input a list of non-orthophoto drone images (NonOrthoimageList);

[0018] The NonOrthoimageList contains NonNumber non-orthophoto angle UAV images;

[0019] S102, use the image segmentation model SAM to segment the Orthoimage and obtain a list of ortho-angle UAV image segmentation results, OrthoSegList;

[0020] The OrthoSegList contains a segmentation blocks SegmentX;

[0021] S103, For each item in OrthoSegList, call the Multimodal Large Language Model (MLLM) to generate text annotations describing the content of the current SegmentX, forming the text content list OrthoSegTextList corresponding to OrthoSegList;

[0022] S104, call the text embedding neural network TEN. For an element in OrthoSegTextList, TEN converts the element in OrthoSegTextList into a one-dimensional semantic vector VectorX. TEN processes each element of OrthoSegTextList to form a semantic vector list OrthoVectorList.

[0023] The dimension of the TEN transformation output vector VectorX is VecDim;

[0024] S105, initialize a loop variable MultiViewCounter=0;

[0025] S106, retrieve the multi-viewcounter image NonOrthoimageX from NonOrthoimageList;

[0026] S107, For the extracted image NonOrthoimageX, use SAM to segment the image NonOrthoimageX and obtain a list of segmentation results NonXSegList;

[0027] The NonXSegList contains b segment blocks SegmentZ;

[0028] S108, for each item in NonXSegList, call MLLM to generate text annotations describing the content of SegmentZ, and finally form the text content list NonXSegTextList corresponding to NonXSegList;

[0029] S109, For an element in NonXSegTextList, TEN converts the element in NonXSegTextList into a one-dimensional vector VectorZ. TEN processes each element of NonXSegTextList to form a semantic vector list NonXVectorList.

[0030] S110, link NonXSegList, NonXSegTextList and NonXVectorList to NonOrthoimageX;

[0031] S111, let MultiViewCounter=MultiViewCounter+1;

[0032] S112, if MultiViewCounter is less than NonNumber, go to S106; otherwise go to S113;

[0033] Wherein, NonNumber is the total number of images in NonOrthoimageList;

[0034] S113 outputs OrthoSegList, NonXSegList, OrthoVectorList, and NonXVectorList.

[0035] Furthermore, in S2, the Region Approximate Neighborhood Module (RPNM) is used to obtain the region approximate neighborhood representation RepVectorList using the semantic vector InputVector, the target non-orthophoto angle UAV image NonOrthoimageY, NonXSegList, and NonXVectorList. Specifically:

[0036] S201, Establish the RPNM module. The input of the RPNM module is a semantic vector InputVector and a non-orthophoto UAV image of the target non-orthoimageY.

[0037] S202, the RPNM module sets the side length and movement step of the square sliding window SlidingWindow according to the width NonOrthoimageYW of the target NonOrthoimageY;

[0038] S203, starting from the top left corner of NonOrthoimageY, follow Step to traverse the image NonOrthoimageY from left to right and from top to bottom;

[0039] S204, find the NonXSegList associated with NonOrthoimageY. For each window position, obtain the set of segmentation blocks InterSectSegs that intersect with SlidingWindow in NonXSegList, and find the semantic vector corresponding to each SegmentZ in NonXVectorList in InterSectSegs, forming a list InterSectVectors. Calculate the mean of all vectors in all InterSectVectors, obtain the mean expression vector MeanPositionVector at the current position, and form a tuple consisting of MeanPositionVector and NonXVectorList at each position.

[0040] S205, After the SlidingWindow traversal is completed, calculate the cosine similarity between the input InputVector and the MeanPositionVector at all positions;

[0041] S206, find the MeanPositionVector corresponding to the lowest cosine similarity between InputVector and MeanPositionVector at all positions, and find the corresponding NonXVectorList as the RepVectorList representing the region approximation neighborhood based on the corresponding tuple of MeanPositionVector.

[0042] S207, output RepVectorList as the result of RPNM.

[0043] Furthermore, in S202, the RPNM module sets the side length and movement step of the square sliding window SlidingWindow based on the width NonOrthoimageYW of the target NonOrthoimageY, specifically as follows:

[0044] The side length of the SlidingWindow is NonOrthoimageYW / 10; the step size of the SlidingWindow is NonOrthoimageYW / 20.

[0045] Furthermore, the direction link semantic knowledge graph generation module DESLGGM in S3 utilizes OrthoSegList, NonOrthoimageList, OrthoVectorList, and RPNM to create a direction link semantic knowledge graph KG, specifically as follows:

[0046] S301, Create the DESLGGM module. The input of the DESLGGM module is: a segment block SegmentX in OrthoSegList and a non-orthophoto UAV image NonOrthoimageZ in NonOrthoimageList.

[0047] S302, Extract the orientation attribute DirFea and the height attribute HigFea from the metadata of the NonOrthoimageZ image;

[0048] S303, retrieve the one-dimensional semantic vector VectorX corresponding to SegmentX in OrthoVectorList;

[0049] S304, calls the RPNM module defined in S2, takes VectorX as the first input InputVector of RPNM, and takes NonOrthoimageZ as the second input NonOrthoimageY of RPNM;

[0050] S305, Receive the output RepVectorList from the RPNM module;

[0051] S306, check if SegmentX has a related knowledge graph KG. If SegmentX has no related knowledge graph KG, go to S307; otherwise, go to S309.

[0052] S307, create a directional link semantic knowledge graph KG, add a core node CenterPoint to KG, and associate KG with SegmentX;

[0053] S308, add a property to CenterPoint and set the property value to VectorX;

[0054] S309, Locate the core node CenterPoint inside KG;

[0055] S310, extract each element from RepVectorList and create a knowledge graph node corresponding to each element;

[0056] The knowledge graph node attributes are the values ​​of the corresponding elements in the RepVectorList;

[0057] S311, associate the knowledge graph node established in S310 with the CenterPoint, and set two attributes for all associations, with the values ​​of DirFea and HigFea respectively;

[0058] S312 outputs KG as the result of DESLGM.

[0059] Furthermore, in step S4, the knowledge graphs corresponding to all segmented blocks in OrthoSegList are obtained using OrthoSegList, NonOrthoimageList, and DESLGGM. The knowledge graphs corresponding to all segmented blocks in OrthoSegList form a global knowledge graph list, GraphList. Specifically:

[0060] S401, get all OrthoSegLists generated in S1, get the total number of elements in OrthoSegLists OrthoSegListNum;

[0061] S402, initialize the first loop variable OrthoCounter=0;

[0062] S403, retrieve the OrthoCounter-th element SegmentT from OrthoSegList;

[0063] S404, initialize the second loop variable NonOrthoCounter=0;

[0064] S405, retrieve the NonOrthoCounter-th element NonOrthoimageT from NonOrthoimageList;

[0065] S406, introduce the DESLGGM module for processing, and obtain the output KG of the DESLGGM module;

[0066] Use SegmentT as the first input SegmentX of DESLGGM, and NonOrthoimageT as the second input NonOrthoimageZ of DESLGGM;

[0067] S407, set NonOrthoCounter = NonOrthoCounter + 1;

[0068] S408, if NonOrthoCounter is less than NonNumber, go to S405; otherwise, go to S409.

[0069] Wherein, NonNumber is the total number of elements in NonOrthoimageList;

[0070] S409, let OrthoCounter=OrthoCounte+1;

[0071] S410, if OrthoCounter is less than OrthoSegListNum, go to S403; otherwise, go to S411.

[0072] Where OrthoSegListNum is the total number of elements in OrthoSegList;

[0073] S411, retrieve the knowledge graphs corresponding to all segments in OrthoSegList and form a global knowledge graph list GraphList;

[0074] Each element in GraphList is a knowledge graph.

[0075] Furthermore, in S5, the knowledge graph establishment module MEMKG utilizes GraphList and NonOrthoimageList to obtain a structurally consistent representation vector YZVector, specifically as follows:

[0076] S501, Create the MEMKG module. The input of the MEMKG module is a knowledge graph GraphY in GraphList;

[0077] S502, Establish a structurally consistent representation vector YZVector;

[0078] The YZVector is a vector containing VecDim A one-dimensional vector with NonNumber elements, all of which have a default value of 0;

[0079] S503, retrieve the GraphY core node CenterPoint;

[0080] S504, retrieve the CenterPointVector property of CenterPoint;

[0081] S505, initialize the loop variable KGCounter=0;

[0082] S506, retrieve the KGCounter-th element NodeX associated with CenterPoint in GraphY;

[0083] S507, retrieve the associated properties DirFea and HigFea from CenterPoint to NodeX;

[0084] S508, retrieve the attribute value NodeXVector of NodeX;

[0085] S509, based on the values ​​of DirFea and HigFea, find the NonXth image in NonOrthoimageList;

[0086] The NonXth image metadata in NonOrthoimageList stores orientation and height information as DirFea and HigFea, respectively.

[0087] Among them, NonX is the label for non-orthogonal angle UAV images;

[0088] S510 uses CenterPoint and NodeXVector to calculate the temporary vector TempVector;

[0089] S511, add TempVector to the NonXth node of YZVector. VecDim to (NonX+1) The element positions of VecDim-1;

[0090] S512, let KGCounter = KGCounter + 1;

[0091] S513, if KGCounter is less than the number of nodes associated with CenterPoint in GraphY, go to S506; otherwise, go to S514.

[0092] S514, normalize YZVector to obtain the normalized YZVector;

[0093] S515, set YZVector = normalized YZVector, and output YZVector as the result of MEMKG.

[0094] Furthermore, the calculation of the temporary vector TempVector using CenterPoint and NodeXVector in S510 specifically involves:

[0095] TempVector=abs(CenterPoint-NodeXVector)

[0096] Where abs is used to calculate the absolute value.

[0097] Further, in S6, the clustering model KMMode is constructed. KMMode calls MEMKG to process the graph elements in GraphList, obtains structurally consistent representation vectors YZVector, associates the graph elements in GraphList with YZVector, and clusters the YZVectors associated with all graph elements in GraphList to obtain the clustering results, specifically:

[0098] S601, Initialize the loop variable ExpressCounter used for graph information representation = 0;

[0099] S602, retrieve the ExpressCounter-th graph element GraphX ​​from GraphList;

[0100] S603 uses MEMKG to process GraphX ​​and obtains a structurally consistent representation vector YZVector;

[0101] S604, associate YZVector with GraphX;

[0102] S605, let ExpressCounter=ExpressCounter+1;

[0103] S606, if ExpressCounter is less than the number of elements in GraphList, go to S602; otherwise go to S607.

[0104] S607 uses the K-Means algorithm to cluster the YZVector associated with all graph elements in GraphList to obtain the K-Means clustering results.

[0105] Furthermore, in S7, the clustering model KMMode is applied to the orthoimage of the UAV at an orthophoto angle and the list NonOrthoimageList consisting of a set of UAV images at non-orthophoto angles to generate an unsupervised classification result image, specifically:

[0106] S701, Create a blank clustering result image ResultImage with the same size as Orthoimage;

[0107] S702, initialize the annotation loop variable ResultCounter=0;

[0108] S703, retrieve the ResultCounter-th element SegmentR from OrthoSegList;

[0109] S704, retrieve the ResultCounter-th element GraphR from GraphList;

[0110] S705, extract the structurally consistent representation vector YZVector associated with GraphR;

[0111] S706, use KMMode to obtain the class target label LableR corresponding to YZVector;

[0112] S707, find all cell positions of SegmentR, and mark all corresponding cells on ResultImage as LabelR;

[0113] S708, let ResultCounter=ResultCounter+1;

[0114] S709, if ResultCounter is less than the number of elements in OrthoSegList, go to S703; otherwise, go to S710.

[0115] S710 outputs ResultImage as the unsupervised classification result image.

[0116] The beneficial effects of this invention are as follows:

[0117] This invention centers on orthophoto-perspective UAV imagery, semantically linking UAV imagery information from multiple perspectives to form a directional link semantic knowledge graph. Through a directional embedding module of the knowledge graph, it integrates information from multiple angles into a more complete ground content information vector, and clusters these vectors to achieve unsupervised classification of UAV imagery. This invention integrates semantic information rather than specific image content during the information introduction stage, effectively utilizing information from multiple angles while avoiding errors caused by significant differences in image content. Furthermore, the introduction of a knowledge graph fully leverages its fault tolerance; discrepancies in individual directions or locations do not significantly alter the knowledge graph attributes. This invention associates and complements the semantic features of ground features observed from different directions, rather than performing physical pixel alignment, thus avoiding the difficulties of ground feature classification in practical work, maximizing the preservation of original perspective information, preventing error propagation, and improving the accuracy of UAV imagery ground feature classification. Attached Figure Description

[0118] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0119] Specific implementation method one: as follows Figure 1As shown, the specific process of the unsupervised classification method for UAV images integrating multi-angle information in this embodiment is as follows:

[0120] S1, Input multi-angle UAV images, segment each image to obtain the corresponding segmentation block, thus obtaining the segmentation result list OrthoSegList for orthogonal angle UAV images and the segmentation result list NonXSegList for non-orthogonal angle UAV images. Convert each segmentation block into a feature vector with semantic information to obtain the semantic vector lists OrthoVectorList and NonXVectorList, specifically:

[0121] S101, Input a single orthophoto drone image (Orthoimage), and input a list of non-orthophoto drone images (NonOrthoimageList);

[0122] The NonOrthoimageList is a list containing multiple non-orthophoto drone images, with the number of elements being NonNumber;

[0123] S102, introduce the image segmentation model SAM (Segment Anything Model) to segment the Orthoimage and obtain a list of segmentation results of the ortho-angle UAV image OrthoSegList;

[0124] The OrthoSegList is a list containing a segment blocks SegmentX;

[0125] The specific steps for introducing SAM to segment Orthoimage are as follows:

[0126] 1) Obtain the number of initial horizontal segmentation points: OrthoimageInitHX = width of Orthoimage / 10;

[0127] 2) Obtain the initial number of points for vertical segmentation: OrthoimageInitZX = height of Orthoimage / 10;

[0128] 3) Forming OrthoimageInitHX OrthoimageInitZX is the initial point matrix for segmentation, where the initial points of the matrix are evenly placed on the Orthoimage.

[0129] 4) SAM segments the Orthoimage based on the initial segmentation point matrix to obtain a segmentation blocks SegmentX, thereby obtaining a segmentation result list OrthoSegList;

[0130] S103. For each item in OrthoSegList, call the Multi-modal Large Language Model (MLLM) to generate text annotations describing the content of SegmentX, and finally form the text content list OrthoSegTextList corresponding to OrthoSegList.

[0131] S104, call the Text Embedding Network (TEN). For an element in OrthoSegTextList, TEN converts the element in OrthoSegTextList into a one-dimensional semantic vector VectorX. TEN processes each element of OrthoSegTextList to form a semantic vector list OrthoVectorList.

[0132] The dimension of the TEN transformation output vector VectorX is VecDim;

[0133] S105, initialize a loop variable MultiViewCounter=0;

[0134] S106, retrieve the multi-viewcounter image NonOrthoimageX from NonOrthoimageList;

[0135] S107, For the extracted image NonOrthoimageX, use SAM to segment the image NonOrthoimageX to obtain a list of segmentation results NonXSegList. NonXSegList is a list containing b segmentation blocks SegmentZ.

[0136] The specific steps for introducing SAM segmentation into NonOrthoimageX are as follows:

[0137] 1) Obtain the number of initial horizontal segmentation points: NonOrthoimageXInitHX = width of NonOrthoimageX / 10;

[0138] 2) Obtain the initial number of points for vertical segmentation: NonOrthoimageXInitZX = height of NonOrthoimageX / 10;

[0139] 3) Formation of ONonOrthoimageXInitHX The initial point matrix for segmentation of NonOrthoimageXInitZX, with the initial points of the matrix evenly placed on NonOrthoimageX;

[0140] 4) SAM segments the Orthoimage based on the initial segmentation point matrix to obtain b segmentation blocks SegmentZ, thus obtaining a segmentation result list NonXSegList;

[0141] S108, for each item in NonXSegList, call MLLM to generate text annotations describing the content of SegmentZ, and finally form the text content list NonXSegTextList corresponding to NonXSegList;

[0142] S109, For an element in NonXSegTextList, TEN converts the element in NonXSegTextList into a one-dimensional vector VectorZ. TEN processes each element of NonXSegTextList to form a semantic vector list NonXVectorList.

[0143] S110, link NonXSegList, NonXSegTextList and NonXVectorList to NonOrthoimageX, so that the corresponding segmentation block, text and semantic vector list of NonOrthoimageX can be found through NonOrthoimageX;

[0144] S111, let MultiViewCounter=MultiViewCounter+1;

[0145] S112, if MultiViewCounter is less than NonNumber, go to S106; otherwise go to S113;

[0146] Wherein, NonNumber is the total number of images in NonOrthoimageList;

[0147] S113 outputs OrthoSegList, NonXSegList, OrthoVectorList, and NonXVectorList.

[0148] This step involves segmenting the input orthophoto drone imagery and a set of non-orthophoto drone imagery, and then using a multimodal large language model and embedding network to convert each segmented block in the imagery into a feature vector with semantic information, providing a foundation for subsequent cross-view semantic association.

[0149] S2, establish the Regional Proximity Neighborhood Module (RPNM). RPNM utilizes the semantic vector InputVector, the target non-orthophoto UAV imagery NonOrthoimageY, NonXSegList, and NonXVectorList to obtain the regional proximity neighborhood representative RepVectorList, specifically:

[0150] S201, Establish the RPNM module. The input of the RPNM module is a semantic vector InputVector and a non-orthophoto UAV image of the target non-orthoimageY.

[0151] S202, the RPNM module sets a fixed-size square sliding window SlidingWindow based on the width NonOrthoimageYW of the target NonOrthoimageY. The side length of SlidingWindow is NonOrthoimageYW / 10; the value of the movement step of SlidingWindow is set to NonOrthoimageYW / 20.

[0152] S203, starting from the top left corner of the target NonOrthoimageY, traverse the image NonOrthoimageY from left to right and from top to bottom according to Step;

[0153] S204, find the NonXSegList associated with NonOrthoimageY. For each window position, obtain the set of segmentation blocks InterSectSegs that intersect with SlidingWindow in NonXSegList, and find the semantic vector corresponding to each SegmentZ in NonXVectorList in InterSectSegs, forming a list InterSectVectors. Calculate the mean of all vectors in all InterSectVectors, obtain the mean expression vector MeanPositionVector at the current position, and form a tuple consisting of MeanPositionVector and NonXVectorList at each position.

[0154] S205, After the SlidingWindow traversal is completed, calculate the cosine similarity between the input InputVector and the MeanPositionVector at all positions;

[0155] S206, find the MeanPositionVector that corresponds to the lowest cosine similarity between InputVector and MeanPositionVector at all positions (the lowest value indicates the most similarity), and find the corresponding NonXVectorList as the RepVectorList representing the region approximation neighborhood based on the corresponding tuple of MeanPositionVector.

[0156] S207, output RepVectorList as the result of RPNM.

[0157] This step defines and executes a core cross-view search module that, based on the input vector, finds the semantically most relevant neighboring regions and objects in a specified multi-view image using non-geometric semantic registration.

[0158] S3. Establish the Directional Element Semantic Linkage Graph Generation Module (DESLGGM). DESLGGM utilizes OrthoSegList, NonOrthoimageList, OrthoVectorList, and the RPNM module to create the directional link semantic knowledge graph KG, specifically:

[0159] S301, Establish the DESLGGM module. The DESLGGM module contains two inputs: one is a segment block SegmentX in OrthoSegList, and the other is a non-orthophoto UAV image NonOrthoimageZ in NonOrthoimageList.

[0160] S302, Extract the orientation attribute DirFea and the height attribute HigFea from the metadata of the NonOrthoimageZ image;

[0161] S303, retrieve the one-dimensional semantic vector VectorX corresponding to SegmentX in OrthoVectorList;

[0162] S304, calls the RPNM module defined in S2, takes VectorX as the first input InputVector of RPNM, and takes NonOrthoimageZ as the second input NonOrthoimageY of RPNM;

[0163] S305, Receive the output RepVectorList from the RPNM module;

[0164] S306, check if SegmentX has a related knowledge graph KG. If SegmentX has no related knowledge graph KG, go to S307; otherwise, go to S309.

[0165] S307, create a directional link semantic knowledge graph KG, add a core node CenterPoint to KG, and associate KG with SegmentX;

[0166] S308, add a property to CenterPoint and set the property value to VectorX;

[0167] S309, Locate the core node CenterPoint inside KG;

[0168] S310, extract each element from RepVectorList and create a knowledge graph node corresponding to each element;

[0169] The node attributes of the knowledge graph are the values ​​of the corresponding elements in the RepVectorList;

[0170] S311, associate the knowledge graph node established in S310 with CenterPoint, and set two attributes for all associations, with the values ​​of DirFea and HigFea respectively;

[0171] S312 outputs KG as the result of DESLGM.

[0172] This step takes a segmentation block on the orthophoto as the core and connects it with semantically related elements from other perspectives to form a graph structure that represents the semantic relationships of multiple perspectives.

[0173] S4, using the OrthoSegList, NonOrthoimageList, and DESLGGM modules, retrieves the knowledge graphs corresponding to all segmented blocks in OrthoSegList. The knowledge graphs corresponding to all segmented blocks in OrthoSegList form a global knowledge graph list, GraphList. Specifically:

[0174] S401, get all OrthoSegLists generated in S1, get the total number of elements in OrthoSegLists OrthoSegListNum;

[0175] S402, initialize the first loop variable OrthoCounter=0;

[0176] S403, retrieve the OrthoCounter-th element SegmentT from OrthoSegList;

[0177] S404, initialize the second loop variable NonOrthoCounter=0;

[0178] S405, retrieve the NonOrthoCounter-th element NonOrthoimageT from NonOrthoimageList;

[0179] S406, introduce the DESLGGM module for processing, and obtain the output KG of the DESLGGM module;

[0180] Use SegmentT as the first input SegmentX of DESLGGM, and NonOrthoimageT as the second input NonOrthoimageZ of DESLGGM;

[0181] S407, set NonOrthoCounter = NonOrthoCounter + 1;

[0182] S408, if NonOrthoCounter is less than NonNumber, go to S405; otherwise, go to S409.

[0183] Wherein, NonNumber is the total number of elements in NonOrthoimageList;

[0184] S409, let OrthoCounter=OrthoCounte+1;

[0185] S410, if OrthoCounter is less than OrthoSegListNum, go to S403; otherwise, go to S411.

[0186] Where OrthoSegListNum is the total number of elements in OrthoSegList;

[0187] S411, retrieve the knowledge graphs corresponding to all segments in OrthoSegList and form a global knowledge graph list GraphList;

[0188] Each element in GraphList is a knowledge graph.

[0189] This step involves repeatedly calling the directional element semantic attachment graph generation module to construct a comprehensive knowledge graph that integrates all multi-angle image information for each segmentation block on the orthophoto, ultimately forming a list containing all the graphs.

[0190] S5, establish the multi-directional embedding module MEMKG of knowledge graph. MEMKG uses GraphList and NonOrthoimageList to obtain the structurally consistent representation vector YZVector, specifically:

[0191] S501, Create the MEMKG module. The input of the MEMKG module is a knowledge graph GraphY in GraphList;

[0192] S502, Establish a structurally consistent representation vector YZVector;

[0193] The YZVector is a vector containing VecDim A one-dimensional vector with NonNumber elements, all of which have a default value of 0;

[0194] S503, retrieve the GraphY core node CenterPoint;

[0195] S504, retrieve the CenterPointVector property of CenterPoint;

[0196] S505, initialize the loop variable KGCounter=0;

[0197] S506, retrieve the KGCounter-th element NodeX associated with CenterPoint in GraphY;

[0198] S507, retrieve the associated properties DirFea and HigFea from CenterPoint to NodeX;

[0199] S508, retrieve the attribute value NodeXVector of NodeX;

[0200] S509, based on the values ​​of DirFea and HigFea, find the NonXth image in NonOrthoimageList;

[0201] The NonXth image metadata in NonOrthoimageList stores orientation and height information as DirFea and HigFea, respectively.

[0202] Among them, NonX is the label for non-orthogonal angle UAV images;

[0203] S510, calculate the temporary vector TempVector=abs(CenterPoint-NodeXVector);

[0204] Where abs is used to calculate the absolute value;

[0205] S511, add TempVector to the NonXth node of YZVector. VecDim to (NonX+1) The element positions of VecDim-1;

[0206] S512, let KGCounter = KGCounter + 1;

[0207] S513, if KGCounter is less than the number of nodes associated with CenterPoint in GraphY, go to S506; otherwise, go to S514.

[0208] S514, normalize YZVector to obtain the normalized YZVector;

[0209] S515, set YZVector = normalized YZVector, and output YZVector as the result of MEMKG.

[0210] This step can integrate the content of the knowledge graph to form a structurally consistent expression vector.

[0211] S6. Construct the clustering model KMMode. KMMode calls MEMKG to process the graph elements in GraphList, obtaining structurally consistent representation vectors YZVector. It then associates the graph elements in GraphList with YZVectors, clusters the YZVectors associated with all graph elements in GraphList, and obtains the clustering results. Specifically:

[0212] S601, Initialize the loop variable ExpressCounter used for graph information representation = 0;

[0213] S602, retrieve the ExpressCounter-th graph element GraphX ​​from GraphList;

[0214] S603 uses MEMKG to process GraphX ​​and obtains a structurally consistent representation vector YZVector;

[0215] S604, associates YZVector with GraphX;

[0216] S605, let ExpressCounter=ExpressCounter+1;

[0217] S606, if ExpressCounter is less than the number of elements in GraphList, go to S602; otherwise go to S607.

[0218] S607 uses the K-Means algorithm to cluster the YZVector associated with all graph elements in GraphList to obtain the K-Means clustering results.

[0219] This step, based on the multi-directional embedding and clustering analysis of the knowledge graph list, clusters all elements of the global knowledge graph list to form a clustering model.

[0220] S7. Apply the KMMode clustering model to the orthoimage of the UAV and the list NonOrthoimageList consisting of a set of non-orthoimages to generate the final unsupervised classification result image, specifically:

[0221] S701, Create a blank clustering result image ResultImage with the same size as Orthoimage;

[0222] S702, initialize the annotation loop variable ResultCounter=0;

[0223] S703, retrieve the ResultCounter-th element SegmentR from OrthoSegList;

[0224] S704, retrieve the ResultCounter-th element GraphR from GraphList;

[0225] S705, extract the structurally consistent representation vector YZVector associated with GraphR;

[0226] S706, use KMMode to judge YZVector and output the judgment result as the target label LabelR;

[0227] S707, find all cell positions of SegmentR, and mark all corresponding cells on ResultImage as LabelR;

[0228] S708, let ResultCounter=ResultCounter+1;

[0229] S709, if ResultCounter is less than the number of elements in OrthoSegList, go to S703; otherwise, go to S710.

[0230] S710 outputs the ResultImage as the final unsupervised classification result image.

Claims

1. An unsupervised classification method for UAV imagery integrating multi-angle information, characterized in that... The specific process of the method is as follows: S1, Input multi-angle UAV images, segment each image to obtain the corresponding segmentation block, thus obtaining the segmentation result list OrthoSegList for orthogonal angle UAV images and the segmentation result list NonXSegList for non-orthogonal angle UAV images. Convert each segmentation block into a feature vector with semantic information to obtain the semantic vector lists OrthoVectorList and NonXVectorList, specifically: The multi-angle drone imagery includes: an orthoimage of a drone at an orthogonal angle and a list of non-orthoimages, NonOrthoimageList. S101, Input a single orthophoto drone image (Orthoimage), Input a list of non-orthophoto drone images (NonOrthoimageList); The NonOrthoimageList contains NonNumber non-orthophoto angle UAV images; S102, use the image segmentation model SAM to segment the Orthoimage and obtain a list of ortho-angle UAV image segmentation results, OrthoSegList; The OrthoSegList contains a segmentation blocks SegmentX; S103, For each item in OrthoSegList, call the Multimodal Large Language Model (MLLM) to generate text annotations describing the content of the current SegmentX, forming the text content list OrthoSegTextList corresponding to OrthoSegList; S104, call the text embedding neural network TEN. For an element in OrthoSegTextList, TEN converts the element in OrthoSegTextList into a one-dimensional semantic vector VectorX. TEN processes each element of OrthoSegTextList to form a semantic vector list OrthoVectorList. The dimension of the TEN transformation output vector VectorX is VecDim; S105, initialize a loop variable MultiViewCounter = 0; S106, retrieve the multi-viewcounter image NonOrthoimageX from NonOrthoimageList; S107, For the extracted image NonOrthoimageX, use SAM to segment the image NonOrthoimageX and obtain a list of segmentation results NonXSegList; The NonXSegList contains b segment blocks SegmentZ; S108, for each item in NonXSegList, call MLLM to generate text annotations describing the content of SegmentZ, and finally form the text content list NonXSegTextList corresponding to NonXSegList; S109, For an element in NonXSegTextList, TEN converts the element in NonXSegTextList into a one-dimensional vector VectorZ. TEN processes each element of NonXSegTextList to form a semantic vector list NonXVectorList. S110, link NonXSegList, NonXSegTextList and NonXVectorList to NonOrthoimageX; S111, let MultiViewCounter=MultiViewCounter+1; S112, if MultiViewCounter is less than NonNumber, go to S106; otherwise go to S113; Wherein, NonNumber is the total number of images in NonOrthoimageList; S113, output OrthoSegList, NonXSegList, OrthoVectorList, and NonXVectorList; S2, establish the Region Approximation Neighborhood Module (RPNM). RPNM utilizes the semantic vector InputVector, the target non-orthophoto UAV imagery NonOrthoimageY, NonXSegList, and NonXVectorList to obtain the region approximation neighborhood representative RepVectorList, specifically: S201, Establish the RPNM module. The input of the RPNM module is a semantic vector InputVector and a non-orthophoto UAV image of the target non-orthoimageY. S202, the RPNM module sets the side length and movement step of the square sliding window SlidingWindow according to the width NonOrthoimageYW of the target NonOrthoimageY; S203, starting from the top left corner of NonOrthoimageY, follow Step to traverse the image NonOrthoimageY from left to right and from top to bottom; S204, find the NonXSegList associated with NonOrthoimageY. For each window position, obtain the set of segmentation blocks InterSectSegs that intersect with SlidingWindow in NonXSegList, and find the semantic vector corresponding to each SegmentZ in NonXVectorList in InterSectSegs, forming a list InterSectVectors. Calculate the mean of all vectors in all InterSectVectors, obtain the mean expression vector MeanPositionVector at the current position, and form a tuple consisting of MeanPositionVector and NonXVectorList at each position. S205, After the SlidingWindow traversal is completed, calculate the cosine similarity between the input InputVector and the MeanPositionVector at all positions; S206, find the MeanPositionVector corresponding to the lowest cosine similarity between InputVector and MeanPositionVector at all positions, and find the corresponding NonXVectorList as the RepVectorList representing the region approximation neighborhood based on the corresponding tuple of MeanPositionVector. S207, output RepVectorList as the result of RPNM; S3, establish the directional link semantic knowledge graph generation module DESLGGM. DESLGGM uses OrthoSegList, NonOrthoimageList, OrthoVectorList and RPNM to create the directional link semantic knowledge graph KG, specifically: S301, Create the DESLGGM module. The input of the DESLGGM module is: a segment block SegmentX in OrthoSegList and a non-orthophoto UAV image NonOrthoimageZ in NonOrthoimageList. S302, Extract the orientation attribute DirFea and the height attribute HigFea from the metadata of the NonOrthoimageZ image; S303, retrieve the one-dimensional semantic vector VectorX corresponding to SegmentX in OrthoVectorList; S304, calls the RPNM module defined in S2, takes VectorX as the first input InputVector of RPNM, and takes NonOrthoimageZ as the second input NonOrthoimageY of RPNM; S305, Receive the output RepVectorList from the RPNM module; S306, check if SegmentX has a related knowledge graph KG. If SegmentX has no related knowledge graph KG, go to S307; otherwise, go to S309. S307, create a directional link semantic knowledge graph KG, add a core node CenterPoint to KG, and associate KG with SegmentX; S308, add a property to CenterPoint and set the property value to VectorX; S309, Locate the core node CenterPoint inside KG; S310, extract each element from RepVectorList and create a knowledge graph node corresponding to each element; The knowledge graph node attributes are the values ​​of the corresponding elements in the RepVectorList; S311, associate the knowledge graph node established in S310 with the CenterPoint, and set two attributes for all associations, with the values ​​of DirFea and HigFea respectively; S312 outputs KG as the result of DESLGM; S4, using OrthoSegList, NonOrthoimageList, and DESLGGM, retrieves the knowledge graphs corresponding to all segments in OrthoSegList. The knowledge graphs corresponding to all segments in OrthoSegList form a global knowledge graph list, GraphList. Specifically: S401, get all OrthoSegLists generated in S1, get the total number of elements in OrthoSegLists OrthoSegListNum; S402, initialize the first loop variable OrthoCounter = 0; S403, retrieve the OrthoCounter-th element SegmentT from OrthoSegList; S404, initialize the second loop variable NonOrthoCounter = 0; S405, retrieve the NonOrthoCounter-th element NonOrthoimageT from NonOrthoimageList; S406, introduce the DESLGGM module for processing, and obtain the output KG of the DESLGGM module; Use SegmentT as the first input SegmentX of DESLGGM, and NonOrthoimageT as the second input NonOrthoimageZ of DESLGGM; S407, set NonOrthoCounter = NonOrthoCounter + 1; S408, if NonOrthoCounter is less than NonNumber, go to S405; otherwise, go to S409. Wherein, NonNumber is the total number of elements in NonOrthoimageList; S409, let OrthoCounter=OrthoCounte+1; S410, if OrthoCounter is less than OrthoSegListNum, go to S403; otherwise, go to S411. Where OrthoSegListNum is the total number of elements in OrthoSegList; S411, retrieve the knowledge graphs corresponding to all segments in OrthoSegList and form a global knowledge graph list GraphList; Each element in GraphList is a knowledge graph; S5, establish the directional embedding module MEMKG for the knowledge graph. MEMKG uses GraphList and NonOrthoimageList to obtain the structurally consistent representation vector YZVector, specifically: S501, Create the MEMKG module. The input of the MEMKG module is a knowledge graph GraphY in GraphList; S502, Establish a structurally consistent representation vector YZVector; The YZVector is a one-dimensional vector containing VecDim×NonNumber elements, with all element values ​​defaulting to 0. S503, retrieve the GraphY core node CenterPoint; S504, retrieve the CenterPointVector property of CenterPoint; S505, initialize the loop variable KGCounter = 0; S506, retrieve the KGCounter-th element NodeX associated with CenterPoint in GraphY; S507, retrieve the associated properties DirFea and HigFea from CenterPoint to NodeX; S508, retrieve the attribute value NodeXVector of NodeX; S509, based on the values ​​of DirFea and HigFea, find the NonXth image in NonOrthoimageList; The NonXth image metadata in NonOrthoimageList stores orientation and height information as DirFea and HigFea, respectively. Among them, NonX is the label for non-orthogonal angle UAV images; S510 uses CenterPoint and NodeXVector to calculate the temporary vector TempVector; S511, add TempVector to the element positions from NonX×VecDim to (NonX+1)×VecDim-1 of YZVector; S512, let KGCounter = KGCounter + 1; S513, if KGCounter is less than the number of nodes associated with CenterPoint in GraphY, go to S506; otherwise, go to S514. S514, normalize YZVector to obtain the normalized YZVector; S515, let YZVector = normalized YZVector, and output YZVector as the result of MEMKG; S6. Construct a clustering model KMMode. KMMode calls MEMKG to process the graph elements in GraphList, obtains structurally consistent representation vectors YZVector, associates the graph elements in GraphList with YZVector, and clusters the YZVector associated with all graph elements in GraphList to obtain the clustering results. S7. The clustering model KMMode is applied to the orthoimage of the UAV at an orthophoto angle and the list NonOrthoimageList consisting of a set of UAV images at non-orthophoto angles to generate an unsupervised classification result image.

2. The unsupervised classification method for UAV images integrating multi-angle information according to claim 1, characterized in that: The RPNM module in S202 sets the side length and movement step of the square sliding window SlidingWindow based on the width NonOrthoimageYW of the target NonOrthoimageY, specifically as follows: The side length of the SlidingWindow is NonOrthoimageYW / 10; the step size of the SlidingWindow is NonOrthoimageYW / 20.

3. The unsupervised classification method for UAV images integrating multi-angle information according to claim 2, characterized in that: The calculation of the temporary vector TempVector using CenterPoint and NodeXVector in S510 is specifically as follows: TempVector=abs(CenterPoint-NodeXVector) Where abs is used to calculate the absolute value.

4. The unsupervised classification method for UAV images integrating multi-angle information according to claim 3, characterized in that: In step S6, the clustering model KMMode is constructed. KMMode calls MEMKG to process the graph elements in GraphList, obtaining structurally consistent representation vectors YZVector. The graph elements in GraphList are associated with YZVectors, and clustering is performed on the YZVectors associated with all graph elements in GraphList to obtain the clustering results. Specifically: S601, Initialize the loop variable ExpressCounter used for graph information representation = 0; S602, retrieve the ExpressCounter-th graph element GraphX ​​from GraphList; S603 uses MEMKG to process GraphX ​​and obtains a structurally consistent representation vector YZVector; S604, associate YZVector with GraphX; S605, let ExpressCounter=ExpressCounter+1; S606, if ExpressCounter is less than the number of elements in GraphList, go to S602; otherwise go to S607. S607 uses the K-Means algorithm to cluster the YZVector associated with all graph elements in GraphList to obtain the K-Means clustering results.

5. The unsupervised classification method for UAV images integrating multi-angle information according to claim 4, characterized in that: In step S7, the clustering model KMMode is applied to the orthoimage of the UAV at an orthophoto angle and the list NonOrthoimageList, which consists of a set of UAV images at non-orthophoto angles, to generate an unsupervised classification result image, specifically: S701, Create a blank clustering result image ResultImage with the same size as Orthoimage; S702, initialize the annotation loop variable ResultCounter = 0; S703, retrieve the ResultCounter-th element SegmentR from OrthoSegList; S704, retrieve the ResultCounter-th element GraphR from GraphList; S705, extract the structurally consistent representation vector YZVector associated with GraphR; S706, use KMMode to obtain the class target label LableR corresponding to YZVector; S707, find all cell positions of SegmentR, and mark all corresponding cells on ResultImage as LabelR; S708, let ResultCounter=ResultCounter+1; S709, if ResultCounter is less than the number of elements in OrthoSegList, go to S703; otherwise, go to S710. S710 outputs ResultImage as the unsupervised classification result image.

Images