Intelligent interactive remote sensing information extraction method and system combined with visual large model

Abstract

The invention discloses an intelligent interactive remote sensing information extraction method and system combined with a large visual model, and relates to the fields of artificial intelligence technology and remote sensing technology. The method comprises: using a trained MAE model and a trained SAM model to initialize parameters of a general unsupervised semantic segmentation network, marking a plurality of first prompt information on a preprocessed remote sensing image to be extracted; constructing a prompt information sampling grid for the preprocessed remote sensing image to be extracted, obtaining a mask corresponding to each second prompt information, cropping the preprocessed remote sensing image to be extracted with each mask as a boundary, obtaining a plurality of slices and grouping them, performing feature mapping and threshold judgment, and determining a final result. The invention can improve the intelligent level of remote sensing interpretation.

Description

Background Art

[0002] High-resolution remote sensing images can provide rich spatial information, which is of great significance for the extraction of typical remote sensing objects, natural resource supervision and urban management. In the past 20 years, the classification of objects based on remote sensing images has gone through several stages, specifically:

[0003] 1) The first stage: Traditional remote sensing object classification is mainly achieved through manual interpretation technology. This method is widely used and highly accurate, but it is time-consuming and labor-intensive, and its application effect is limited by prior knowledge.

[0004] 2) The second stage: With the development of multispectral / hyperspectral data, scholars have tried to use spectral indices to extract specific ground object information and combine it with manual interpretation to achieve remote sensing ground object classification. However, this method still requires manual setting of extraction features and thresholds, and the method of using pixels as calculation units ignores the spatial correlation of remote sensing objects to a certain extent.

[0005] 3) The third stage: Traditional machine learning methods such as support vector machines and random forests combined with object-oriented technology have become the mainstream technology for remote sensing object classification. On the one hand, they fully utilize the spatial spectrum information of remote sensing, and on the other hand, they solve the problem of artificial threshold setting to a certain extent. However, the recognition accuracy of this type of segmentation method with "object" as the target often depends on the accuracy of the initial multi-scale segmentation. Therefore, it is not effective for recognition scenarios dominated by linear objects.

[0006] 4) The fourth stage: With the rapid development of artificial intelligence, machine learning methods represented by deep learning have made significant progress in the field of remote sensing applications. When there are sufficient samples, the accuracy and efficiency are significantly better than traditional machine learning methods. With the rapid development of Transformer models in the field of machine vision, using the advantages of large models to extract remote sensing information has become a new trend. However, remote sensing multi-class target recognition based on deep learning technology still faces challenges, specifically:

[0007] 1) Remote sensing data has multimodal properties. Different spatial resolutions, phases, and sensors lead to variable application scenarios. The application effect of deep learning methods based on supervised classification is often limited by the number and richness of samples. How to build on the advantages of pre-trained large models and reduce the model's dependence on samples is the main technical problem;

[0008] 2) Although significant progress has been made in the application of deep learning technology in remote sensing multi-target extraction, its accuracy is still difficult to meet application requirements, and manual interpretation is still the norm. How to combine deep learning with traditional manual interpretation and propose an intelligent interactive interpretation method to solve the "last mile" problem of deep learning technology application is the difficulty of current research. Summary of the invention

[0009] The technical problem to be solved by the present invention is to reduce the dependence on samples and improve the extraction accuracy, etc., and provide an intelligent interactive remote sensing information extraction method and system combined with a large visual model to reduce the dependence on samples and improve the extraction accuracy of remote sensing information, as follows:

[0010] 1) In the first aspect, the present invention provides an intelligent interactive remote sensing information extraction method combined with a large visual model, and the specific technical scheme is as follows:

[0011] S1. Construct a general unsupervised semantic segmentation network, use the trained MAE model and the trained SAM model to initialize the parameters of the general unsupervised semantic segmentation network, preprocess the remote sensing image to be extracted, visualize the preprocessed remote sensing image to be extracted, mark multiple first prompt information on the preprocessed remote sensing image to be extracted, construct a prompt information sampling grid for the preprocessed remote sensing image to be extracted, use the parameter-initialized general unsupervised semantic segmentation network to infer any second prompt information and the preprocessed remote sensing image to be extracted, and obtain a mask corresponding to the second prompt information, until a mask corresponding to each second prompt information is obtained, wherein the image coordinates of each intersection in the prompt information sampling grid are respectively a second prompt information;

[0012] S2, using each mask as a boundary, cropping the pre-processed remote sensing image to be extracted to obtain multiple slices;

[0013] S3, dividing the slices with the first prompt information into a target group, and dividing the slices without the first prompt information into a background group;

[0014] S4. Based on the CLIP model, feature mapping is performed on the target group and the background group, the similarity between each mask in the target group and the background group is calculated, and the mask with a similarity greater than the set threshold is extracted as the final result.

[0015] The beneficial effects of the intelligent interactive remote sensing information extraction method combined with a large visual model provided by the present invention are as follows:

[0016] The advantages of large semantic segmentation models such as the trained MAE model and the trained SAM model are used to realize unsupervised feature extraction of remote sensing images. The remote sensing intelligent interaction framework is constructed in combination with the first prompt information, and the data-driven and knowledge-driven methods are cleverly integrated. The target group and the background group are feature mapped based on the CLIP model, and then the feature matching technology and threshold judgment are used to realize the automatic extraction of remote sensing multi-target information to obtain the final result. The extraction accuracy can be guaranteed, which solves the "last mile" problem of deep learning technology in the application of remote sensing interpretation. It is of great significance to improve the intelligent level of remote sensing interpretation and carry out wide-area recognition applications.

[0017] On the basis of the above scheme, the intelligent interactive remote sensing information extraction method combined with a large visual model of the present invention can also be improved as follows.

[0018] Furthermore, it also includes:

[0019] The first prompt information is added to the missed-recognition target and the mis-recognition target, and the process returns to execute S3 to correct the final result.

[0020] Furthermore, the remote sensing image to be extracted is preprocessed, including:

[0021] The outliers in the remote sensing image to be extracted are removed, and the remote sensing image to be extracted after the outliers are removed is normalized to obtain the preprocessed remote sensing image to be extracted.

[0022] Further, the pre-processed remote sensing image to be extracted is visualized, and a plurality of first prompt information is marked on the pre-processed remote sensing image to be extracted, including:

[0023] The preprocessed remote sensing image to be extracted is visualized, and multiple positive sample points are selected on the foreground of the preprocessed remote sensing image to be extracted through visual interpretation, and multiple negative sample points are selected on the background of the preprocessed remote sensing image to be extracted through visual interpretation. The information of any sample point includes: the map coordinates of the sample point, and whether the sample point is located on the foreground or the background. The information of each sample point is a first prompt information.

[0024] 2) In the second aspect, the present invention also provides an intelligent interactive remote sensing information extraction system combined with a large visual model, and the specific technical solution is as follows:

[0025] It includes a construction module, a first reasoning module, a second reasoning module, a cutting module, a dividing module and an extracting module;

[0026] The construction module is used to: construct a general unsupervised semantic segmentation network, and use the trained MAE model and the trained SAM model to initialize the parameters of the general unsupervised semantic segmentation network;

[0027] The first reasoning module is used to: preprocess the remote sensing image to be extracted, visualize the preprocessed remote sensing image to be extracted, and mark a plurality of first prompt information on the preprocessed remote sensing image to be extracted;

[0028] The second reasoning module is used to: construct a prompt information sampling grid for the preprocessed remote sensing image to be extracted, and use the general unsupervised semantic segmentation network after parameter initialization to reason on any second prompt information and the preprocessed remote sensing image to be extracted to obtain a mask corresponding to the second prompt information, until a mask corresponding to each second prompt information is obtained, wherein the image coordinates of each intersection in the prompt information sampling grid are respectively a second prompt information;

[0029] The cropping module is used to: crop the pre-processed remote sensing image to be extracted with each mask as the boundary to obtain multiple slices;

[0030] The dividing module is used to: divide the slices with the first prompt information into a target group, and divide the slices without the first prompt information into a background group;

[0031] The extraction module is used to: perform feature mapping on the target group and the background group based on the CLIP model, calculate the similarity between each mask in the target group and the background group, and extract the mask with a similarity greater than a set threshold as the final result.

[0032] Based on the above scheme, the intelligent interactive remote sensing information extraction system combined with a large visual model of the present invention can also be improved as follows.

[0033] Furthermore, a correction module is included, and the correction module is used to:

[0034] The first prompt information is added to the missed targets and the misidentified targets, and the segmentation module and the extraction module are re-called to correct the final result.

[0035] Furthermore, the first reasoning module is also specifically used for:

[0036] The outliers in the remote sensing image to be extracted are removed, and the remote sensing image to be extracted after the outliers are removed is normalized to obtain the preprocessed remote sensing image to be extracted.

[0037] Furthermore, the first reasoning module is also specifically used for:

[0038] The preprocessed remote sensing image to be extracted is visualized, and multiple positive sample points are selected on the foreground of the preprocessed remote sensing image to be extracted through visual interpretation, and multiple negative sample points are selected on the background of the preprocessed remote sensing image to be extracted through visual interpretation. The information of any sample point includes: the map coordinates of the sample point, and whether the sample point is located on the foreground or the background. The information of each sample point is a first prompt information.

[0039] 3) In a third aspect, the present invention further provides a computer device, comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor, so that the computer device implements any one of the above-mentioned intelligent interactive remote sensing information extraction methods combined with a large visual model.

[0040] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor so that the computer implements any of the above-mentioned intelligent interactive remote sensing information extraction methods combined with a large visual model.

[0041] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0043] Figure 1 A schematic diagram of a flow chart of an intelligent interactive remote sensing information extraction method combined with a large visual model according to an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a mask corresponding to each second prompt information is obtained;

[0045] Figure 3 A schematic diagram to obtain the final result;

[0046] Figure 4 A schematic diagram of the structure of an intelligent interactive remote sensing information extraction system combined with a large visual model according to an embodiment of the present invention;

[0047] Figure 5 The figure is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0049] like Figure 1 As shown, an intelligent interactive remote sensing information extraction method combined with a visual large model according to an embodiment of the present invention comprises the following steps:

[0050] S1. Construct a general unsupervised semantic segmentation network, use the trained MAE model and the trained SAM model to initialize the parameters of the general unsupervised semantic segmentation network, preprocess the remote sensing image to be extracted, visualize the preprocessed remote sensing image to be extracted, mark multiple first prompt information on the preprocessed remote sensing image to be extracted, construct a prompt information sampling grid for the preprocessed remote sensing image to be extracted, use the general unsupervised semantic segmentation network after parameter initialization to infer any second prompt information and the preprocessed remote sensing image to be extracted, and obtain the mask corresponding to the second prompt information, until the mask corresponding to each second prompt information is obtained, wherein the surface coordinates of each intersection in the prompt information sampling grid are respectively a second prompt information.

[0051] Among them, a general unsupervised semantic segmentation network is constructed, and the parameters of the general unsupervised semantic segmentation network are initialized using the trained MAE model and the trained SAM model. Specifically:

[0052] In the Pytorch environment, build a general unsupervised semantic segmentation network, such as Figure 2 As shown, the general unsupervised semantic segmentation network includes an image encoder and an image decoder, specifically:

[0053] 1) The parameters of the image encoder are initialized from the trained MAE model. The image encoder is VisionTransformer (ViT). The present invention preferentially uses the ViT-H / 16 version of ViT, which mainly includes an upsampling layer, a Patch feature embedding layer, and four isometric global attention modules. Figure 2 The Transformer module in is the isometric global attention module, which can better capture the long-distance dependencies between image slices. The output of the image encoder is the image feature embedding information, which is used to input the image decoder.

[0054] 2) The image decoder includes a ViT module and two layers of MLP, which are used to restore the image feature embedding information into label data of the same size as the input image. The parameters of the image decoder are initialized from the trained SAM model.

[0055] 3) In addition to receiving the image embedding features output by the image encoder, namely the image feature embedding information, the image decoder will also receive prompt information consisting of coordinate points or coordinate boxes, which specifically refers to the second prompt information. The second prompt information will be input into the keyword prompter and mapped into a feature vector and then passed into the image decoder.

[0056] The input image of the image encoder forms an image of size 1024×1024 through an upsampling layer and a Padding operation, and through the patch embedding operation of the Patch feature embedding layer and four equidistant global attention modules, 16×16 image feature embedding information is obtained and input into the image encoder. By leveraging the performance advantages of large models such as MAE and SAM trained on a vast amount of natural images, the general unsupervised semantic segmentation network is enabled to have the ability to perform semantic segmentation under zero-shot conditions.

[0057] Among them, the SAM model is the SAM visual large model launched by Meta, and its full English name is: SegmentAnything Model.

[0058] Among them, the remote sensing image to be extracted is preprocessed, the preprocessed remote sensing image to be extracted is visualized, and multiple first prompt information is marked on the preprocessed remote sensing image to be extracted, specifically including S10 to S11.

[0059] S10. Preprocess the remote sensing image to be extracted:

[0060] Outliers in the remote sensing image to be extracted are removed, and the remote sensing image to be extracted after removing outliers is normalized, specifically including S100 to S101.

[0061] S100. Use the first formula to statistically analyze the pixel values in the remote sensing image to be extracted, and remove the outliers in the part where the quantile in the data is greater than 99%. The first formula is: Q represents the 99% quantile in D, N represents the number of pixel values in D, that is, the number of pixels, and D represents all pixel values in the remote sensing image to be extracted.

[0062] S101. Use the second formula to normalize the remote sensing image to be extracted after removing outliers within 0-255. The second formula is: x∈[D<Q], [D<Q] represents all pixel values in D that are less than Q, x is any pixel value in D that is less than Q, that is, any pixel value in [D<Q], x max represents the maximum value in [D<Q], x min represents the minimum value in [D<Q], X ′ represents the data after normalizing x.

[0063] S11. Perform visualization and mark multiple first prompt information, including:

[0064] The preprocessed remote sensing image to be extracted is visualized, and multiple positive sample points are selected on the foreground of the preprocessed remote sensing image to be extracted through visual interpretation, and multiple negative sample points are selected on the background of the preprocessed remote sensing image to be extracted through visual interpretation. The information of any sample point includes: the map coordinates of the sample point, and whether the sample point is located on the foreground or the background. The information of each sample point is a first prompt information.

[0065] Among them, the positive sample point is recorded as Point pos , the negative sample point is recorded as Point nag , each sample point includes three pieces of information (p x ,p y, t), p x The horizontal coordinate of the sample point’s surface coordinate, p y The ordinate of the sample point's coordinates on the image, t represents the type of the sample point, that is, t represents whether the sample point is located in the foreground or the background. The t of a positive sample point located in the foreground is 1, and the t of a negative sample point located in the background is 0. The information of each sample point is a first prompt information.

[0066] Among them, a prompt information sampling grid is constructed for the preprocessed remote sensing image to be extracted, and a general unsupervised semantic segmentation network with initialized parameters is used to infer any second prompt information and the preprocessed remote sensing image to be extracted to obtain a mask corresponding to the second prompt information, until a mask corresponding to each second prompt information is obtained, wherein the map coordinates of each intersection in the prompt information sampling grid are respectively a second prompt information, specifically including S1000 to S1001.

[0067] S1000, calculating the spacing of the prompt information sampling grid, specifically including S10000 to S10001:

[0068] S10000, calculate the texture complexity of the remote sensing image to be extracted after preprocessing:

[0069] The third formula is used to calculate the second-order moment of the image histogram of the preprocessed remote sensing image to be extracted as the texture complexity of the preprocessed remote sensing image to be extracted. The third formula is:

[0070]

[0071] Among them, k p is the texture complexity of the remote sensing image to be extracted after preprocessing, p iis the i-th pixel value of the remote sensing image to be extracted after preprocessing, h(P) is the image histogram of the remote sensing image to be extracted after preprocessing, L is the number of different pixel value levels of the remote sensing image to be extracted after preprocessing, i is a positive integer, and i is not greater than the number of all pixel values ​​less than Q in D, and m represents the average value of all pixel values ​​of the remote sensing image to be extracted after preprocessing.

[0072] S10001. Calculate the spacing:

[0073] K p Normalization is performed to obtain k with a value range of 0-1 n , calculate the spacing s of the prompt information sampling grid according to the fourth formula, the fourth formula is: Its value needs to be rounded down, R represents the length value of the long side or wide side of the remote sensing image to be extracted after preprocessing. When R represents the length value of the long side of the remote sensing image to be extracted after preprocessing, the calculated s is sl, sl represents the spacing of the prompt information sampling grid of the long side of the remote sensing image to be extracted after preprocessing. When R represents the length value of the wide side of the remote sensing image to be extracted after preprocessing, the calculated s is sd, sd represents the spacing of the prompt information sampling grid of the wide side of the remote sensing image to be extracted after preprocessing. The prompt information sampling grid is constructed according to sl and sd, and the surface coordinates of each intersection of the prompt information sampling grid are respectively a second prompt information.

[0074] S1001, obtaining a mask corresponding to each second prompt information:

[0075] Assume that the hint information sampling grid constructed for the preprocessed remote sensing image to be extracted contains n second hint information, use the general unsupervised semantic segmentation network after parameter initialization to infer any second hint information and the preprocessed remote sensing image to be extracted, and obtain the mask corresponding to the second hint information. When a second hint information is input, multiple masks are obtained. Preferably, only the mask with the highest reasoning accuracy is retained as the mask corresponding to the second hint information, and only the result with the highest reasoning accuracy is retained among the multiple masks corresponding to each second hint information as the mask corresponding to each second hint information, and finally a set M of masks corresponding to each second hint information is obtained. n , which is the instance segmentation result, and the fifth formula is used for reasoning. The fifth formula is:

[0076]

[0077] Points j Represents the jth second prompt information, where j represents a positive integer, Img represents: the remote sensing image to be extracted after preprocessing, Model sam {Img,(Point j)} means Points j The general unsupervised semantic segmentation network after initialization with Img input parameters is obtained. j The corresponding mask, It means that the n highest precision mask data, i.e., the mask corresponding to each second prompt information, are superimposed in the channel dimension. In other words, Points j The corresponding masks are stored in the set M n In the above example, all the second prompt information are traversed to obtain a set M of masks corresponding to each second prompt information. n .

[0078] S2. Using each mask as a boundary, the preprocessed remote sensing image to be extracted is cropped to obtain multiple slices.

[0079] S3: Divide the slices with the first prompt information into a target group, and divide the slices without the first prompt information into a background group.

[0080] S4, based on the CLIP model, feature mapping is performed on the target group and the background group, the similarity between each mask in the target group and the background group is calculated, and the mask with a similarity greater than the set threshold is extracted as the final result, such as Figure 3 As shown, S3 to S4 are explained as follows, specifically including S40 to S41:

[0081] S40, the obtained M n The pre-processed remote sensing data is clipped using each mask in as the boundary to obtain the remote sensing data slice combination G n , G n The data slices with the first prompt information and the data slices without the first prompt information are divided into the target group G n1 and background group G n0 , the CLIP model with pre-trained weights is used to perform feature mapping on the two sets of slice data to obtain high-dimensional spaces, namely F n1 and F n0 , see the sixth formula for details, then, based on F n1 and F n0 , and use feature matching technology to estimate F n1 The similarity between the mean of and each mask in the background group, the similarity being specifically cosine similarity, is specifically calculated using the seventh formula.

[0082] Among them, the sixth formula is: (F n1, F n0 )=Model CLIP (G n1 ,G n0 ), ModelCLIP (G n1 ,G n0 ) means: G n1 Input the CLIP model with pre-trained weights and get F n1 , and G n0 Input the CLIP model with pre-trained weights and get F n0 The seventh formula is: F n ′ Indicates F n1 The mean value, V k Represents the kth mask in the background group, k is a positive integer, ∥F n ′ ∥ represents vector F n ′ The modulus of ∥V k ∥ means V k The model, <F n ′ ,V k >

[0083] Represents the vector F n ′ and vector V k The inner product of Meaning: Calculate F n ′ and V k The cosine similarity between n ′ and V k The cosine similarity between n1 The similarity between the mean of and the k-th mask, S n Indicates F n1 The set of similarities between the mean of and each mask in the background group,

[0084] Indicates: F n1 The similarity between the mean of and the kth mask is stored in the set S n In the above example, we traverse all the masks and get the n1 The set S of similarities between the mean of and each mask in the background group n .

[0085] S41, extract S n The mask with similarity greater than the set threshold is taken as the final result. By adjusting the threshold value, the tolerance of feature differences of different targets in the extraction results can be increased or decreased to achieve better extraction effect. The threshold can be set according to the actual situation.

[0086] Among them, the CLIP model is the CLIP multimodal large model launched by OpenAI, and its full English name is: Contrastive Language-Image Pretraining.

[0087] Optionally, in the above technical solution, it also includes:

[0088] S5. Add the first prompt information to the missed targets and the misidentified targets, and return to execute S3 to correct the final result. Specifically:

[0089] For some unsatisfactory extracted targets, i.e., missed targets and misidentified targets, the final result can be corrected twice by adding the first prompt information, specifically:

[0090] Add positive sample points to missed targets, add negative sample points to misidentified targets, bring the updated first prompt information into S3, re-execute S3 and S4, complete the correction of the final result, and further improve the extraction accuracy.

[0091] The present invention is an intelligent interactive remote sensing information extraction method combined with a large visual model, which uses the advantages of a semantic segmentation large model to achieve unsupervised feature extraction of remote sensing data, combines coordinate point prompt information to build a remote sensing intelligent interactive framework, and realizes automatic extraction of remote sensing multi-target information through feature similarity estimation technology based on CLIP. The present invention realizes the extraction of similar remote sensing targets by proposing a new feature matching technology, and cleverly integrates data-driven and knowledge-driven methods by building an interactive framework of deep learning algorithms and manual interpretation, solving the "last mile" problem of deep learning technology in the application of remote sensing interpretation, which is of great significance for improving the intelligent level of remote sensing interpretation and carrying out wide-area recognition applications.

[0092] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0093] like Figure 4 As shown, an intelligent interactive remote sensing information extraction system 200 combined with a visual large model according to an embodiment of the present invention includes a construction module 201, a first reasoning module 202, a second reasoning module 203, a clipping module 204, a division module 205 and an extraction module 206;

[0094] The construction module 201 is used to: construct a general unsupervised semantic segmentation network, and use the trained MAE model and the trained SAM model to initialize the parameters of the general unsupervised semantic segmentation network;

[0095] The first reasoning module 202 is used to: preprocess the remote sensing image to be extracted, visualize the preprocessed remote sensing image to be extracted, and mark a plurality of first prompt information on the preprocessed remote sensing image to be extracted;

[0096] The second reasoning module 203 is used to: construct a prompt information sampling grid for the preprocessed remote sensing image to be extracted, and use the general unsupervised semantic segmentation network after parameter initialization to reason on any second prompt information and the preprocessed remote sensing image to be extracted to obtain a mask corresponding to the second prompt information, until a mask corresponding to each second prompt information is obtained, wherein the image coordinates of each intersection in the prompt information sampling grid are respectively a second prompt information;

[0097] The cropping module 204 is used to: crop the pre-processed remote sensing image to be extracted with each mask as a boundary to obtain multiple slices;

[0098] The dividing module 205 is used to: divide the slices with the first prompt information into a target group, and divide the slices without the first prompt information into a background group;

[0099] The extraction module 206 is used to perform feature mapping on the target group and the background group based on the CLIP model, calculate the similarity between each mask in the target group and the background group, and extract the mask with a similarity greater than a set threshold as the final result.

[0100] Optionally, in the above technical solution, a correction module is further included, and the correction module is used to:

[0101] The first prompt information is added to the missed targets and the misidentified targets, and the division module 205 and the extraction module 206 are re-called to correct the final result.

[0102] Optionally, in the above technical solution, the first reasoning module 202 is further specifically used for:

[0103] The outliers in the remote sensing image to be extracted are removed, and the remote sensing image to be extracted after the outliers are removed is normalized to obtain the preprocessed remote sensing image to be extracted.

[0104] Optionally, in the above technical solution, the first reasoning module 202 is further specifically used for:

[0105] The preprocessed remote sensing image to be extracted is visualized, and multiple positive sample points are selected on the foreground of the preprocessed remote sensing image to be extracted through visual interpretation, and multiple negative sample points are selected on the background of the preprocessed remote sensing image to be extracted through visual interpretation. The information of any sample point includes: the map coordinates of the sample point, and whether the sample point is located on the foreground or the background. The information of each sample point is a first prompt information.

[0106] It should be noted that the beneficial effects of the intelligent interactive remote sensing information extraction system 200 combined with a visual large model provided in the above embodiment are the same as the beneficial effects of the intelligent interactive remote sensing information extraction method combined with a visual large model, which will not be repeated here. In addition, when the system provided in the above embodiment realizes its functions, it only takes the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0107] like Figure 5 As shown, a computer device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the computer device 300 implements any of the above-mentioned intelligent interactive remote sensing information extraction methods combined with a large visual model, specifically:

[0108] The computer device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the computer device 300 implements the intelligent interactive remote sensing information extraction method combined with a large visual model provided in the above embodiment. Of course, the computer device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the computer device 300 may also include other components for realizing the functions of the device, which will not be repeated here.

[0109] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above-mentioned intelligent interactive remote sensing information extraction methods combined with a large visual model.

[0110] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0111] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes any of the above-mentioned intelligent interactive remote sensing information extraction methods combined with a large visual model.

[0112] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0113] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0114] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0115] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An intelligent interactive remote sensing information extraction method combined with a large visual model, characterized in that: include: S1. Construct a general unsupervised semantic segmentation network, use the trained MAE model and the trained SAM model to initialize the parameters of the general unsupervised semantic segmentation network, preprocess the remote sensing image to be extracted, visualize the preprocessed remote sensing image to be extracted, mark multiple first prompt information on the preprocessed remote sensing image to be extracted, construct a prompt information sampling grid for the preprocessed remote sensing image to be extracted, use the general unsupervised semantic segmentation network after parameter initialization to infer any second prompt information and the preprocessed remote sensing image to be extracted, and obtain a mask corresponding to the second prompt information, until a mask corresponding to each second prompt information is obtained, wherein the image coordinates of each intersection in the prompt information sampling grid are respectively a second prompt information; The preprocessed remote sensing image to be extracted is visualized, and a plurality of first prompt information is marked on the preprocessed remote sensing image to be extracted, including: Visualizing the preprocessed remote sensing image to be extracted, selecting a plurality of positive sample points on the foreground of the preprocessed remote sensing image to be extracted by visual interpretation, and selecting a plurality of negative sample points on the background of the preprocessed remote sensing image to be extracted by visual interpretation, wherein the information of any sample point includes: the drawing coordinates of the sample point, and whether the sample point is located on the foreground or the background, and the information of each sample point is a first prompt information; A prompt information sampling grid is constructed for the preprocessed remote sensing image to be extracted, and a general unsupervised semantic segmentation network after parameter initialization is used to infer any second prompt information and the preprocessed remote sensing image to be extracted, so as to obtain a mask corresponding to the second prompt information, until a mask corresponding to each second prompt information is obtained, wherein the image coordinates of each intersection point in the prompt information sampling grid are respectively a second prompt information, specifically including S1000 to S1001: S1000, calculating the spacing of the prompt information sampling grid, specifically including S10000 to S10001: S10000, calculating the texture complexity of the remote sensing image to be extracted after preprocessing; S10001. Calculate the spacing: Will Normalization is performed to obtain a value range of 0-1 , calculate the spacing s of the prompt information sampling grid according to the fourth formula, the fourth formula is: , its value needs to be rounded down, R represents the length of the long side or wide side of the remote sensing image to be extracted after preprocessing, is the texture complexity of the remote sensing image to be extracted after preprocessing; S1001, obtaining a mask corresponding to each second prompt information: Assume that the hint information sampling grid constructed for the preprocessed remote sensing image to be extracted contains n second hint information, use the general unsupervised semantic segmentation network after parameter initialization to infer any second hint information and the preprocessed remote sensing image to be extracted, and obtain the mask corresponding to the second hint information. When a second hint information is input, multiple masks are obtained. Preferably, only the mask with the highest reasoning accuracy is retained as the mask corresponding to the second hint information, and only the result with the highest reasoning accuracy is retained among the multiple masks corresponding to each second hint information as the mask corresponding to each second hint information, and finally a set of masks corresponding to each second hint information is obtained. , which is the instance segmentation result, and the fifth formula is used for reasoning. The fifth formula is: in, represents the jth second prompt information, where j is a positive integer. Indicates: the remote sensing image to be extracted after preprocessing, Indicates that and The general unsupervised semantic segmentation network after input parameter initialization is obtained The corresponding mask, It represents the channel-dimensional superposition operation of the n highest-precision mask data, i.e., the mask corresponding to each second prompt information, respectively. Indicates that The corresponding masks are stored in the collection In the above example, all the second prompt information are traversed to obtain a set of masks corresponding to each second prompt information. ; S2, taking each mask as a boundary, cropping the preprocessed remote sensing image to be extracted to obtain a plurality of slices; S3, dividing the slices with the first prompt information into a target group, and dividing the slices without the first prompt information into a background group; S4. Feature mapping is performed on the target group and the background group based on the CLIP model, similarity between each mask in the target group and the background group is calculated, and masks with similarity greater than a set threshold are extracted as the final result.

2. The intelligent interactive remote sensing information extraction method combined with a visual large model according to claim 1 is characterized in that: Also includes: The first prompt information is added to the missed-recognition target and the mis-recognition target, and the process returns to execute S3 to correct the final result.

3. The intelligent interactive remote sensing information extraction method combined with a visual large model according to claim 1 or 2, characterized in that: Preprocess the remote sensing images to be extracted, including: Outliers in the remote sensing image to be extracted are removed, and the remote sensing image to be extracted after the outliers are removed is normalized to obtain the preprocessed remote sensing image to be extracted.

4. An intelligent interactive remote sensing information extraction system combined with a large visual model, characterized in that: It includes a construction module, a first reasoning module, a second reasoning module, a cutting module, a dividing module and an extracting module; The construction module is used to: construct a general unsupervised semantic segmentation network, and use the trained MAE model and the trained SAM model to initialize the parameters of the general unsupervised semantic segmentation network; The first reasoning module is used to: preprocess the remote sensing image to be extracted, visualize the preprocessed remote sensing image to be extracted, and mark a plurality of first prompt information on the preprocessed remote sensing image to be extracted; The first reasoning module is further specifically used for: Visualizing the preprocessed remote sensing image to be extracted, selecting a plurality of positive sample points on the foreground of the preprocessed remote sensing image to be extracted by visual interpretation, and selecting a plurality of negative sample points on the background of the preprocessed remote sensing image to be extracted by visual interpretation, wherein the information of any sample point includes: the drawing coordinates of the sample point, and whether the sample point is located on the foreground or the background, and the information of each sample point is a first prompt information; The second reasoning module is used to: construct a prompt information sampling grid for the preprocessed remote sensing image to be extracted, and use the general unsupervised semantic segmentation network initialized with the parameters to reason on any second prompt information and the preprocessed remote sensing image to be extracted to obtain a mask corresponding to the second prompt information, until a mask corresponding to each second prompt information is obtained, wherein the image coordinates of each intersection in the prompt information sampling grid are respectively a second prompt information; The cropping module is used to: crop the pre-processed remote sensing image to be extracted with each mask as a boundary to obtain multiple slices; The division module is used to: divide the slices with the first prompt information into a target group, and divide the slices without the first prompt information into a background group; The extraction module is used to: perform feature mapping on the target group and the background group based on the CLIP model, calculate the similarity between each mask in the target group and the background group, and extract the mask with a similarity greater than a set threshold as the final result; A prompt information sampling grid is constructed for the preprocessed remote sensing image to be extracted, and a general unsupervised semantic segmentation network after parameter initialization is used to infer any second prompt information and the preprocessed remote sensing image to be extracted, so as to obtain a mask corresponding to the second prompt information, until a mask corresponding to each second prompt information is obtained, wherein the image coordinates of each intersection point in the prompt information sampling grid are respectively a second prompt information, specifically including S1000 to S1001: S1000, calculating the spacing of the prompt information sampling grid, specifically including S10000 to S10001: S10000, calculating the texture complexity of the remote sensing image to be extracted after preprocessing; S10001. Calculate the spacing: Will Normalization is performed to obtain a value range of 0-1 , calculate the spacing s of the prompt information sampling grid according to the fourth formula, the fourth formula is: , its value needs to be rounded down, R represents the length of the long side or wide side of the remote sensing image to be extracted after preprocessing, is the texture complexity of the remote sensing image to be extracted after preprocessing; S1001, obtaining a mask corresponding to each second prompt information: Assume that the hint information sampling grid constructed for the preprocessed remote sensing image to be extracted contains n second hint information, use the general unsupervised semantic segmentation network after parameter initialization to infer any second hint information and the preprocessed remote sensing image to be extracted, and obtain the mask corresponding to the second hint information. When a second hint information is input, multiple masks are obtained. Preferably, only the mask with the highest reasoning accuracy is retained as the mask corresponding to the second hint information, and only the result with the highest reasoning accuracy is retained among the multiple masks corresponding to each second hint information as the mask corresponding to each second hint information, and finally a set of masks corresponding to each second hint information is obtained. , which is the instance segmentation result, and the fifth formula is used for reasoning. The fifth formula is: in, represents the jth second prompt information, where j is a positive integer. Indicates: the remote sensing image to be extracted after preprocessing, Indicates that and The general unsupervised semantic segmentation network after input parameter initialization is obtained The corresponding mask, It represents the channel-dimensional superposition operation of the n highest-precision mask data, i.e., the mask corresponding to each second prompt information, respectively. Indicates that The corresponding masks are stored in the collection In the above example, all the second prompt information are traversed to obtain a set of masks corresponding to each second prompt information. .

5. The intelligent interactive remote sensing information extraction system combined with a visual large model according to claim 4 is characterized in that: It also includes a correction module, the correction module is used to: The first prompt information is added to the missed-recognition target and the mis-recognition target, and the division module and the extraction module are re-called to correct the final result.

6. The intelligent interactive remote sensing information extraction system combined with a visual large model according to claim 4 or 5, characterized in that: The first reasoning module is further specifically used for: Outliers in the remote sensing image to be extracted are removed, and the remote sensing image to be extracted after the outliers are removed is normalized to obtain the preprocessed remote sensing image to be extracted.

7. A computer device, characterized in that: The computer device includes a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the computer device implements the intelligent interactive remote sensing information extraction method combined with a visual large model as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer implements the intelligent interactive remote sensing information extraction method combined with a large visual model as described in any one of claims 1 to 3.

Images