Remote sensing image prediction method and device and terminal equipment

Abstract

The invention is suitable for the technical field of remote sensing images, and provides a remote sensing image prediction method and device and terminal equipment, and the method comprises the steps: inputting a plurality of patch blocks of a remote sensing image into a preset Vit network for feature coding, outputting a visual feature vector and a category feature vector, inputting the visual feature vector into a salient region sensing module, and outputting a salient region sensing module; the method comprises the steps of generating a salient region feature vector, inputting a category feature vector into a multi-instance learning module, generating label probability distribution, generating a visual attention context feature vector in combination with a region attention mechanism and the salient region feature vector, and generating a semantic attention context feature vector in combination with a label attention mechanism and the label probability distribution. And finally, fusing the visual attention context feature vector and the semantic attention context feature vector to generate a fused feature and generate a prediction result of the remote sensing image. Therefore, the accuracy of remote sensing image prediction classification can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of remote sensing images, and in particular, relates to a remote sensing image prediction method, device and terminal equipment. Background Art

[0002] As a joint technology of image understanding and language generation, high-resolution remote sensing image description has attracted the attention of more and more researchers. Specifically, the high-resolution remote sensing image description task can generate a comprehensive and appropriate natural language based on the content of the image. Due to the novelty and creativity of this technology, high-resolution remote sensing image description has certain application prospects in civil and military fields, including human-computer interaction, blind assistants, battlefield environment analysis, etc.

[0003] In the related art, the current prediction and classification algorithms for remote sensing images have the problem of poor accuracy. Summary of the invention

[0004] The embodiments of the present application provide a remote sensing image prediction method, apparatus, and terminal device, which can solve the problem of poor accuracy of current remote sensing image prediction and classification algorithms.

[0005] In the first aspect, an embodiment of the present application provides a remote sensing image prediction method, including: inputting multiple patch blocks corresponding to the remote sensing image into a preset Vit network for feature encoding, and outputting a visual feature vector and a category feature vector corresponding to the remote sensing image; inputting the visual feature vector into a salient area perception module to generate a salient area feature vector; inputting the category feature vector into a multi-instance learning module to generate a label probability distribution; combining the regional attention mechanism and the salient area feature vector to generate a visual attention context feature vector; combining the label attention mechanism and the label probability distribution to generate a semantic attention context feature vector; fusing the visual attention context feature vector and the semantic attention context feature vector through an adaptive fusion layer to generate a fusion feature, and generating a prediction result of the remote sensing image based on the fusion feature.

[0006] In a possible implementation manner of the first aspect, the salient region perception module is composed of a single-layer Transformer and a region positioning module, and the visual feature vector is input into the salient region perception module to generate the salient region feature vector, including:

[0007] Input the visual feature vector into a single-layer Transformer to generate the final attention output and the feature vector of the selected region;

[0008] The final attention output and the feature vector of the selected region are input into the region localization module to generate a salient region feature vector.

[0009] Optionally, in another possible implementation of the first aspect, the step of inputting the visual feature vector into a single-layer Transformer to generate a final attention output and a feature vector of a region to be selected includes:

[0010] Based on the query vector Q of the visual feature vector v , key vector K v and the value vector V v Combined with the multi-head attention mechanism, the final attention output is generated as follows:

[0011]

[0012] S = Concat(s1,...,s H )

[0013] Among them, s i is the similarity matrix of the i-th attention head, Q vi is the query vector of the ith attention head, is the key vector of the ith attention head, d is the feature dimension, S is the final attention output, and s H is the similarity matrix of the Hth attention head;

[0014] According to the final attention output and the value vector of the visual feature vector, the output value of the single-layer Transformer is generated, and the feature vector of the selected region is determined according to the output value of the single-layer Transformer, as follows:

[0015] V s =AddNorm(FFN(AddNorm(SV v )))

[0016] V s =[C s ,v s1 ,v s2 ,...,v sm ]

[0017] V I =[v s1 ,v s2 ,...,v sm ]

[0018] Among them, V s is the output value of a single-layer Transformer, S is the final attention output, V v is the value vector of the visual feature vector, C s is the global feature of the salient region, V sm is the local feature of the mth region in the salient region, V I is the feature vector of the region to be selected.

[0019] Optionally, in another possible implementation of the first aspect, the step of inputting the final attention output and the feature vector of the candidate region into a region positioning module to generate a salient region feature vector includes:

[0020] The similarity matrix of the i-th attention head is split, and the similarity mean of all attention heads is generated according to the similarity matrix of the i-th attention head after splitting, as follows:

[0021] s i =[s i(0) ,s i(1) ,...,s i(m) ]

[0022] s′ i =[s i(1) ,...,s i(m) ]

[0023]

[0024] Among them, s i(0) is the similarity between categories, s′ i is the similarity between categories and regions, and S′ is the mean similarity of all attention heads;

[0025] Sort the weight values ​​corresponding to the mean similarity values ​​of all attention heads and filter out the patch indexes of the top K significant regions;

[0026] Map the patch indexes of the first K salient regions to the feature vectors of the candidate regions to generate the salient region feature vector V R .

[0027] Optionally, in another possible implementation manner of the first aspect, the step of inputting the category feature vector into the multi-instance learning module to generate a label probability distribution includes:

[0028]

[0029] Where P is the label probability distribution, is the Sigmoid activation function, W a is the weight matrix used to aggregate information in the multi-instance learning network, W c is the weight matrix of the category feature vector, C is the category feature vector, A l is the instance feature vector of the salient region.

[0030] Optionally, in another possible implementation of the first aspect, the above-mentioned combining the regional attention mechanism and the salient region feature vector to generate a visual attention context feature vector includes:

[0031] Determine the salient region feature vector as the visual attention feature vector;

[0032] According to the visual attention feature vector, a visual attention context feature vector is generated as follows:

[0033]

[0034] Among them, R is the visual attention feature vector, is the visual attention context feature vector, as well as They represent the query vector, key vector, and value vector of the visual attention feature vector respectively.

[0035] Optionally, in another possible implementation of the first aspect, the above-mentioned combining the label attention mechanism and the label probability distribution to generate a semantic attention context feature vector includes:

[0036] Determine the label probability distribution as the semantic attention feature vector;

[0037] According to the semantic attention feature vector, a semantic attention context feature vector is generated as follows:

[0038]

[0039] Among them, L is the semantic attention feature vector, is the semantic attention context feature vector, as well as They represent the query vector, key vector, and value vector of the semantic attention feature vector respectively.

[0040] Optionally, in another possible implementation of the first aspect, the visual attention context feature vector and the semantic attention context feature vector are fused through the adaptive fusion layer to generate a fusion feature, and a prediction result of the remote sensing image is generated according to the fusion feature, including:

[0041]

[0042]

[0043]

[0044] Among them, μ is the fusion coefficient, W μ is the trainable weight matrix, is the fusion feature, It is the prediction result of remote sensing image.

[0045] In the second aspect, an embodiment of the present application provides a prediction device for a remote sensing image, comprising: an output module, used to input multiple patch blocks corresponding to the remote sensing image into a preset Vit network for feature encoding, and output a visual feature vector and a category feature vector corresponding to the remote sensing image; a first generation module, used to input the visual feature vector into a salient area perception module to generate a salient area feature vector; a second generation module, used to input the category feature vector into a multi-instance learning module to generate a label probability distribution; a third generation module, used to combine the regional attention mechanism and the salient area feature vector to generate a visual attention context feature vector; a fourth generation module, used to combine the label attention mechanism and the label probability distribution to generate a semantic attention context feature vector; a fifth generation module, used to fuse the visual attention context feature vector and the semantic attention context feature vector through an adaptive fusion layer to generate a fusion feature, and generate a prediction result of the remote sensing image based on the fusion feature.

[0046] In a possible implementation manner of the second aspect, the salient region perception module is composed of a single-layer Transformer and a region positioning module, and the first generation module includes:

[0047] The first generation unit is used to input the visual feature vector into the single-layer Transformer to generate the final attention output and the feature vector of the selected region;

[0048] The second generating unit is used to input the final attention output and the feature vector of the candidate region into the region positioning module to generate a salient region feature vector.

[0049] Optionally, in a possible implementation manner of the second aspect, the first generating unit is specifically configured to:

[0050] Based on the query vector Q of the visual feature vector v , key vector K v and the value vector V v Combined with the multi-head attention mechanism, the final attention output is generated as follows:

[0051]

[0052] S = Concat(s1,...,s H )

[0053] Among them, s i is the similarity matrix of the i-th attention head, Q vi is the query vector of the ith attention head, is the key vector of the ith attention head, d is the feature dimension, S is the final attention output, and sH is the similarity matrix of the Hth attention head;

[0054] According to the final attention output and the value vector of the visual feature vector, the output value of the single-layer Transformer is generated, and the feature vector of the selected region is determined according to the output value of the single-layer Transformer, as follows:

[0055] V s =AddNorm(FFN(AddNorm(SV v )))

[0056] V s =[C s ,v s1 ,v s2 ,...,v sm ]

[0057] V I =[v s1 ,v s2 ,...,v sm ]

[0058] Among them, V s is the output value of a single-layer Transformer, S is the final attention output, V v is the value vector of the visual feature vector, C s is the global feature of the salient region, V sm is the local feature of the mth region in the salient region, V I is the feature vector of the region to be selected.

[0059] Optionally, in another possible implementation manner of the second aspect, the second generating unit is specifically configured to:

[0060] The similarity matrix of the i-th attention head is split, and the similarity mean of all attention heads is generated according to the similarity matrix of the i-th attention head after splitting, as follows:

[0061] s i =[s i(0) ,s i(1) ,...,s i(m) ]

[0062] s′ i =[s i(1) ,...,s i(m) ]

[0063]

[0064] Among them, s i(0) is the similarity between categories, s′i is the similarity between categories and regions, and S′ is the mean similarity of all attention heads;

[0065] Sort the weight values ​​corresponding to the mean similarity values ​​of all attention heads and filter out the patch indexes of the top K significant regions;

[0066] Map the patch indexes of the first K salient regions to the feature vectors of the candidate regions to generate the salient region feature vector V R .

[0067] Optionally, in another possible implementation manner of the second aspect, the second generating module is specifically as follows:

[0068]

[0069] Where P is the label probability distribution, is the Sigmoid activation function, W a is the weight matrix used to aggregate information in the multi-instance learning network, W c is the weight matrix of the category feature vector, C is the category feature vector, A l is the instance feature vector of the salient region.

[0070] Optionally, in another possible implementation manner of the second aspect, the third generating module includes:

[0071] A first determining unit, configured to determine a salient region feature vector as a visual attention feature vector;

[0072] The third generating unit is used to generate a visual attention context feature vector according to the visual attention feature vector, as follows:

[0073]

[0074] Among them, R is the visual attention feature vector, is the visual attention context feature vector, as well as They represent the query vector, key vector, and value vector of the visual attention feature vector respectively.

[0075] Optionally, in another possible implementation manner of the second aspect, the fourth generating module includes:

[0076] A second determining unit, configured to determine the label probability distribution as a semantic attention feature vector;

[0077] The fourth generating unit is used to generate a semantic attention context feature vector according to the semantic attention feature vector, as follows:

[0078]

[0079] Among them, L is the semantic attention feature vector, is the semantic attention context feature vector, as well as They represent the query vector, key vector, and value vector of the semantic attention feature vector respectively.

[0080] Optionally, in another possible implementation of the second aspect, the fifth generating module is specifically as follows:

[0081]

[0082]

[0083]

[0084] Among them, μ is the fusion coefficient, W μ is the trainable weight matrix, is the fusion feature, It is the prediction result of remote sensing image.

[0085] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the remote sensing image prediction method as described above when executing the computer program.

[0086] In the technical solution of the present application, multiple patch blocks of remote sensing images are input into a preset Vit network for feature encoding, and visual feature vectors and category feature vectors are output. The visual feature vectors are input into a significant region perception module to generate significant region feature vectors, and the category feature vectors are input into a multi-instance learning module to generate label probability distributions. Then, the regional attention mechanism and the significant region feature vectors are combined to generate visual attention context feature vectors, and the label attention mechanism and the label probability distribution are combined to generate semantic attention context feature vectors. Finally, the visual attention context feature vectors and the semantic attention context feature vectors are fused to generate fusion features and generate prediction results of remote sensing images. This can improve the accuracy of remote sensing image prediction and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0088] Figure 1 It is a flowchart of a remote sensing image prediction method provided by an embodiment of the present application;

[0089] Figure 2 is a structural diagram of a salient area perception module provided in an embodiment of the present application;

[0090] Figure 3 It is a schematic diagram of the overall process of a remote sensing image prediction method provided in one embodiment of the present application;

[0091] Figure 4 It is a schematic diagram of a feature fusion process provided by an embodiment of the present application;

[0092] Figure 5 It is a schematic diagram of the flow of a remote sensing image prediction device provided by an embodiment of the present application;

[0093] Figure 6 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0094] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0095] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0096] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0097] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0098] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0099] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0100] The remote sensing image prediction method, apparatus and terminal device provided in the present application are described in detail below with reference to the accompanying drawings.

[0101] Figure 1 A schematic flow chart of a remote sensing image prediction method provided in an embodiment of the present application is shown.

[0102] like Figure 1 As shown, the prediction method of the remote sensing image comprises the following steps:

[0103] Step 101: input multiple patch blocks corresponding to the remote sensing image into a preset Vit network for feature encoding, and output a visual feature vector and a category feature vector corresponding to the remote sensing image.

[0104] It should be noted that when encountering large images in business scenarios, such as remote sensing images, the image size can reach hundreds of megabytes or several gigabytes or even larger. At this time, opening the image and putting it into the memory will be a bit sluggish, and directly putting the original image into GPU inference will not have enough video memory, so the original image needs to be cut into multiple patch blocks of the same size according to the patch size before subsequent inference and prediction.

[0105] In the embodiment of the present application, the pre-trained Vit network processes multiple patch blocks corresponding to the remote sensing image, which can well capture the local and global information of the remote sensing image.

[0106] Step 102: input the visual feature vector into a salient region perception module to generate a salient region feature vector.

[0107] In one embodiment of the present application, Figure 2 As shown in the structural diagram of the salient region perception module, the salient region perception module is composed of a single-layer Transformer and a region positioning module. Further, in an embodiment of the present application, the above step 102 also includes:

[0108] Step 1021, input the visual feature vector into a single-layer Transformer to generate a final attention output and a feature vector of a selected region;

[0109] Furthermore, the above step 1021 includes:

[0110] Based on the query vector Q of the visual feature vector v , key vector K v and the value vector V v Combined with the multi-head attention mechanism, the final attention output is generated as follows:

[0111]

[0112] S = Concat(s1,...,s H )

[0113] Among them, s i is the similarity matrix of the i-th attention head, Q vi is the query vector of the ith attention head, is the key vector of the ith attention head, d is the feature dimension, S is the final attention output, and s H is the similarity matrix of the Hth attention head;

[0114] According to the final attention output and the value vector of the visual feature vector, the output value of the single-layer Transformer is generated, and the feature vector of the selected region is determined according to the output value of the single-layer Transformer, as follows:

[0115] V s =AddNorm(FFN(AddNorm(SV v )))

[0116] V s =[C s ,v s1 ,vs2 ,...,v sm ]

[0117] V I =[v s1 ,v s2 ,...,v sm ]

[0118] Among them, V s is the output value of a single-layer Transformer, S is the final attention output, V v is the value vector of the visual feature vector, C s is the global feature of the salient region, V sm is the local feature of the mth region in the salient region, V I is the feature vector of the region to be selected.

[0119] Step 1022, input the final attention output and the feature vector of the candidate region into the region positioning module to generate a salient region feature vector.

[0120] Furthermore, the above step 1022 includes:

[0121] The similarity matrix of the i-th attention head is split, and the similarity mean of all attention heads is generated according to the similarity matrix of the i-th attention head after splitting, as follows:

[0122] s i =[s i(0) ,s i(1) ,...,s i(m) ]

[0123] s′ i =[s i(1) ,...,s i(m) ]

[0124]

[0125] Among them, s i(0) is the similarity between categories, s′ i is the similarity between categories and regions, and S′ is the mean similarity of all attention heads;

[0126] Sort the weight values ​​corresponding to the mean similarity values ​​of all attention heads and filter out the patch indexes of the top K significant regions;

[0127] Map the patch indexes of the first K salient regions to the feature vectors of the candidate regions to generate the salient region feature vector V R .

[0128] Step 103: input the category feature vector into a multi-instance learning module to generate a label probability distribution.

[0129] In this embodiment, the specific process of step 103 can be expressed by the following formula:

[0130]

[0131] Where P is the label probability distribution, is the Sigmoid activation function, W a is the weight matrix used to aggregate information in the multi-instance learning network, W c is the weight matrix of the category feature vector, C is the category feature vector, A l is the instance feature vector of the salient region.

[0132] Step 104, combining the regional attention mechanism and the salient region feature vector to generate a visual attention context feature vector.

[0133] Furthermore, in an embodiment of the present application, the above step 104 further includes:

[0134] Step 1041, determining the salient region feature vector as the visual attention feature vector;

[0135] Step 1042, generating a visual attention context feature vector according to the visual attention feature vector, as follows:

[0136]

[0137] Among them, R is the visual attention feature vector, is the visual attention context feature vector, as well as They represent the query vector, key vector, and value vector of the visual attention feature vector respectively.

[0138] Step 105, combining the label attention mechanism and the label probability distribution to generate a semantic attention context feature vector.

[0139] Furthermore, in an embodiment of the present application, the above step 105 further includes:

[0140] Step 1051, determining the label probability distribution as a semantic attention feature vector;

[0141] Step 1052, generating a semantic attention context feature vector according to the semantic attention feature vector, as follows:

[0142]

[0143] Among them, L is the semantic attention feature vector, is the semantic attention context feature vector, as well as They represent the query vector, key vector, and value vector of the semantic attention feature vector respectively.

[0144] Step 106, the visual attention context feature vector and the semantic attention context feature vector are fused through an adaptive fusion layer to generate a fusion feature, and a prediction result of the remote sensing image is generated based on the fusion feature.

[0145] In this embodiment, the specific process of step 106 is:

[0146]

[0147]

[0148]

[0149] Among them, μ is the fusion coefficient, W μ is the trainable weight matrix, is the fusion feature, It is the prediction result of remote sensing image.

[0150] Figure 3 What is shown is a schematic diagram of the overall flow of the remote sensing image prediction method of step 101 to step 106 of the present application;

[0151] Figure 4 What is shown is a schematic diagram of the feature fusion process of the present application, which illustrates the basic principle of fusing the visual attention context feature vector and the semantic attention context feature vector through an adaptive fusion layer in step 106 to generate a fused feature.

[0152] The prediction method of remote sensing images provided in the present application first inputs multiple patch blocks of remote sensing images into a preset Vit network for feature encoding, outputs visual feature vectors and category feature vectors, inputs the visual feature vectors into a significant region perception module to generate significant region feature vectors, inputs the category feature vectors into a multi-instance learning module to generate label probability distribution, then combines the regional attention mechanism and the significant region feature vectors to generate visual attention context feature vectors, combines the label attention mechanism and the label probability distribution to generate semantic attention context feature vectors, and finally fuses the visual attention context feature vectors and the semantic attention context feature vectors to generate fusion features and generate the prediction results of remote sensing images. This can improve the accuracy of remote sensing image prediction and classification.

[0153] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0154] Corresponding to the prediction method of remote sensing images in the above embodiment, Figure 5 A structural block diagram of a remote sensing image prediction device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0155] Reference Figure 5 , the device 500 comprises:

[0156] The output module 501 is used to input multiple patch blocks corresponding to the remote sensing image into a preset Vit network for feature encoding, and output a visual feature vector and a category feature vector corresponding to the remote sensing image;

[0157] A first generating module 502, configured to input the visual feature vector into a salient region perception module to generate a salient region feature vector;

[0158] The second generation module 503 is used to input the category feature vector into the multiple instance learning module to generate a label probability distribution;

[0159] The third generation module 504 is used to generate a visual attention context feature vector by combining the regional attention mechanism and the salient region feature vector;

[0160] A fourth generation module 505 is used to generate a semantic attention context feature vector by combining the label attention mechanism and the label probability distribution;

[0161] The fifth generation module 506 is used to fuse the visual attention context feature vector and the semantic attention context feature vector through an adaptive fusion layer to generate a fusion feature, and generate a prediction result of the remote sensing image according to the fusion feature.

[0162] In actual use, the remote sensing image prediction device provided in the embodiment of the present application can be configured in any terminal device to execute the aforementioned remote sensing image prediction method.

[0163] The prediction device of remote sensing images provided by the present application first inputs multiple patch blocks of remote sensing images into a preset Vit network for feature encoding, outputs visual feature vectors and category feature vectors, inputs the visual feature vectors into a significant region perception module to generate significant region feature vectors, inputs the category feature vectors into a multi-instance learning module to generate label probability distribution, then combines the regional attention mechanism and the significant region feature vectors to generate visual attention context feature vectors, combines the label attention mechanism and the label probability distribution to generate semantic attention context feature vectors, and finally fuses the visual attention context feature vectors and the semantic attention context feature vectors to generate fusion features and generate the prediction results of remote sensing images. This can improve the accuracy of remote sensing image prediction and classification.

[0164] In a possible implementation of the embodiment of the present application, the salient region perception module is composed of a single-layer Transformer and a region positioning module, and the first generation module 502 includes:

[0165] The first generation unit is used to input the visual feature vector into the single-layer Transformer to generate the final attention output and the feature vector of the selected region;

[0166] The second generating unit is used to input the final attention output and the feature vector of the candidate region into the region positioning module to generate a salient region feature vector.

[0167] Furthermore, in a possible implementation of the embodiment of the present application, the first generating unit is specifically configured to:

[0168] Based on the query vector Q of the visual feature vector v , key vector K v and the value vector V v Combined with the multi-head attention mechanism, the final attention output is generated as follows:

[0169]

[0170] S = Concat(s1,...,s H )

[0171] Among them, s i is the similarity matrix of the i-th attention head, Q vi is the query vector of the ith attention head, is the key vector of the ith attention head, d is the feature dimension, S is the final attention output, and s H is the similarity matrix of the Hth attention head;

[0172] According to the final attention output and the value vector of the visual feature vector, the output value of the single-layer Transformer is generated, and the feature vector of the selected region is determined according to the output value of the single-layer Transformer, as follows:

[0173] V s =AddNorm(FFN(AddNorm(SV v )))

[0174] V s =[C s ,v s1 ,v s2 ,...,v sm ]

[0175] V I =[v s1 ,v s2 ,...,v sm ]

[0176] Among them, V s is the output value of a single-layer Transformer, S is the final attention output, V v is the value vector of the visual feature vector, C s is the global feature of the salient region, V sm is the local feature of the mth region in the salient region, V I is the feature vector of the region to be selected.

[0177] Furthermore, in another possible implementation of the embodiment of the present application, the second generating unit is specifically configured to:

[0178] The similarity matrix of the i-th attention head is split, and the similarity mean of all attention heads is generated according to the similarity matrix of the i-th attention head after splitting, as follows:

[0179] s i =[s i(0) ,s i(1) ,...,s i(m) ]

[0180] s′ i =[s i(1) ,...,s i(m) ]

[0181]

[0182] Among them, s i(0) is the similarity between categories, s′ i is the similarity between categories and regions, and S′ is the mean similarity of all attention heads;

[0183] Sort the weight values ​​corresponding to the mean similarity values ​​of all attention heads and filter out the patch indexes of the top K significant regions;

[0184] Map the patch indexes of the first K salient regions to the feature vectors of the candidate regions to generate the salient region feature vector V R .

[0185] Furthermore, in another possible implementation of the embodiment of the present application, the second generating module 503 is specifically as follows:

[0186]

[0187] Where P is the label probability distribution, is the Sigmoid activation function, W a is the weight matrix used to aggregate information in the multi-instance learning network, W c is the weight matrix of the category feature vector, C is the category feature vector, A l is the instance feature vector of the salient region.

[0188] Furthermore, in another possible implementation of the embodiment of the present application, the third generating module 504 includes:

[0189] A first determining unit, configured to determine a salient region feature vector as a visual attention feature vector;

[0190] The third generating unit is used to generate a visual attention context feature vector according to the visual attention feature vector, as follows:

[0191]

[0192] Among them, R is the visual attention feature vector, is the visual attention context feature vector, as well as They represent the query vector, key vector, and value vector of the visual attention feature vector respectively.

[0193] Furthermore, in another possible implementation of the embodiment of the present application, the fourth generating module 505 includes:

[0194] A second determining unit, configured to determine the label probability distribution as a semantic attention feature vector;

[0195] The fourth generating unit is used to generate a semantic attention context feature vector according to the semantic attention feature vector, as follows:

[0196]

[0197] Among them, L is the semantic attention feature vector, is the semantic attention context feature vector, as well as They represent the query vector, key vector, and value vector of the semantic attention feature vector respectively.

[0198] Furthermore, in another possible implementation of the embodiment of the present application, the fifth generating module 506 is specifically as follows:

[0199]

[0200]

[0201]

[0202] Among them, μ is the fusion coefficient, W μ is the trainable weight matrix, is the fusion feature, It is the prediction result of remote sensing image.

[0203] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0204] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0205] In order to implement the above embodiments, the present application also proposes a terminal device.

[0206] Figure 6 A schematic diagram of the structure of a terminal device according to an embodiment of the present application.

[0207] like Figure 6 As shown, the terminal device 200 includes:

[0208] The memory 210 and at least one processor 220, a bus 230 connecting different components (including the memory 210 and the processor 220), the memory 210 stores a computer program, and when the processor 220 executes the program, the remote sensing image prediction method described in the embodiment of the present application is implemented.

[0209] Bus 230 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0210] The terminal device 200 typically includes a variety of electronic device readable media, which can be any available media that can be accessed by the terminal device 200, including volatile and non-volatile media, removable and non-removable media.

[0211] The memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. The terminal device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 260 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to bus 230 via one or more data medium interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0212] A program / utility 280 having a set (at least one) of program modules 270 may be stored, for example, in the memory 210, such program modules 270 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 270 generally perform the functions and / or methods of the embodiments described herein.

[0213] The terminal device 200 can also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), and can also communicate with one or more devices that enable users to interact with the terminal device 200, and / or communicate with any device that enables the terminal device 200 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be carried out through an input / output (I / O) interface 292. In addition, the terminal device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 293. As shown in the figure, the network adapter 293 communicates with other modules of the terminal device 200 through a bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the terminal device 200, including but not limited to: microcode, device driver, redundant processing unit, external disk drive array, RAID system, tape drive, and data backup storage system, etc.

[0214] The processor 220 executes various functional applications and data processing by running the programs stored in the memory 210 .

[0215] It should be noted that the implementation process and technical principles of the terminal device of this embodiment refer to the aforementioned explanation of the remote sensing image prediction method of the embodiment of the present application, and will not be repeated here.

[0216] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0217] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0218] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0219] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0220] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0221] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0222] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0223] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A remote sensing image prediction method, characterized in that: include: Input multiple patch blocks corresponding to the remote sensing image into a preset Vit network for feature encoding, and output a visual feature vector and a category feature vector corresponding to the remote sensing image; Inputting the visual feature vector into a salient region perception module to generate a salient region feature vector; Inputting the category feature vector into a multi-instance learning module to generate a label probability distribution; Combining the regional attention mechanism with the salient region feature vector to generate a visual attention context feature vector; Combining the label attention mechanism with the label probability distribution, generating a semantic attention context feature vector; The visual attention context feature vector and the semantic attention context feature vector are fused through an adaptive fusion layer to generate a fusion feature, and a prediction result of the remote sensing image is generated based on the fusion feature.

2. The method according to claim 1, characterized in that The salient region perception module is composed of a single-layer Transformer and a region positioning module. The visual feature vector is input into the salient region perception module to generate a salient region feature vector, including: Input the visual feature vector into the single-layer Transformer to generate a final attention output and a feature vector of a selected region; The final attention output and the feature vector of the candidate region are input into the region positioning module to generate the salient region feature vector.

3. The method according to claim 2, characterized in that The step of inputting the visual feature vector into the single-layer Transformer to generate a final attention output and a feature vector of a selected region includes: According to the query vector Q of the visual feature vector v , key vector K v and the value vector V v And combined with the multi-head attention mechanism, the final attention output is generated, as follows: S=Concat(s1,...,s H ) Among them, s i is the similarity matrix of the i-th attention head, Q vi is the query vector of the ith attention head, is the key vector of the ith attention head, d is the feature dimension, S is the final attention output, s H is the similarity matrix of the Hth attention head; According to the final attention output and the value vector of the visual feature vector, the output value of the single-layer Transformer is generated, and the feature vector of the region to be selected is determined according to the output value of the single-layer Transformer, as follows: V s =AddNorm(FFN(AddNorm(SV v ))) V s =[C s ,v s1 ,v s2 ,...,v sm ] V I =[v s1 ,v s2 ,...,v sm ] Among them, V s is the output value of the single-layer Transfomer, S is the final attention output, V v is the value vector of the visual feature vector, C s is the global feature of the salient region, V sm is the local feature of the mth region in the salient region, V I is the feature vector of the region to be selected.

4. The method according to claim 3, characterized in that The step of inputting the final attention output and the feature vector of the candidate region into the region positioning module to generate the salient region feature vector comprises: The similarity matrix of the i-th attention head is split, and the similarity mean of all attention heads is generated according to the split similarity matrix of the i-th attention head, as follows: s i =[s i(0) ,s i(1) ,...,s i(m) ] s′ i =[s i(1) ,...,s i(m) ] Among them, s i(0) is the similarity between categories, s′ i is the similarity between categories and regions, and S′ is the mean similarity of all attention heads; Sort the weight values ​​corresponding to the mean similarities of all the attention heads, and filter out the patch indexes of the top K significant areas; Map the patch indexes of the first K salient regions to the feature vector of the candidate region to generate the salient region feature vector V R .

5. The method according to claim 1, characterized in that The step of inputting the category feature vector into a multi-instance learning module to generate a label probability distribution includes: Where P is the label probability distribution, σ is the Sigmoid activation function, and W a is the weight matrix used to aggregate information in the multi-instance learning network, W c is the weight matrix of the class feature vector, C is the class feature vector, A l is the instance feature vector of the salient region.

6. The method according to claim 1, characterized in that The combining the regional attention mechanism and the salient region feature vector to generate a visual attention context feature vector includes: Determining the salient region feature vector as a visual attention feature vector; The visual attention context feature vector is generated according to the visual attention feature vector, specifically as follows: Where R is the visual attention feature vector, V l t is the visual attention context feature vector, as well as Respectively represent the query vector, key vector and value vector of the visual attention feature vector.

7. The method according to claim 6, characterized in that The combining of the label attention mechanism and the label probability distribution to generate a semantic attention context feature vector includes: Determining the label probability distribution as a semantic attention feature vector; According to the semantic attention feature vector, the semantic attention context feature vector is generated as follows: Where L is the semantic attention feature vector, is the semantic attention context feature vector, as well as Respectively represent the query vector, key vector and value vector of the semantic attention feature vector.

8. The method according to claim 7, characterized in that The step of fusing the visual attention context feature vector and the semantic attention context feature vector through an adaptive fusion layer to generate a fusion feature, and generating a prediction result of the remote sensing image according to the fusion feature includes: Among them, μ is the fusion coefficient, W μ is the trainable weight matrix, F l t is the fusion feature, is the prediction result of the remote sensing image.

9. A remote sensing image prediction device, characterized in that: include: An output module is used to input multiple patch blocks corresponding to the remote sensing image into a preset Vit network for feature encoding, and output a visual feature vector and a category feature vector corresponding to the remote sensing image; A first generating module, used for inputting the visual feature vector into a salient region perception module to generate a salient region feature vector; A second generation module is used to input the category feature vector into a multiple instance learning module to generate a label probability distribution; A third generation module is used to generate a visual attention context feature vector by combining the regional attention mechanism and the salient region feature vector; A fourth generation module, used to generate a semantic attention context feature vector by combining the label attention mechanism and the label probability distribution; The fifth generation module is used to fuse the visual attention context feature vector and the semantic attention context feature vector through an adaptive fusion layer to generate a fusion feature, and generate a prediction result of the remote sensing image based on the fusion feature.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images