Remote sensing image prediction method and device, terminal equipment and storage medium

Abstract

The invention is suitable for the technical field of remote sensing, and provides a remote sensing image prediction method and device, terminal equipment and a storage medium, and the method comprises the steps: obtaining a primary feature vector of a remote sensing image, carrying out the feature enhancement of the primary feature vector according to a multi-head attention mechanism, and generating an enhanced feature vector; generating an nth implicit strata state vector and a memory vector for thinking the LSTM layer according to the enhanced feature vector, substituting the optimal thinking times into the nth implicit strata state vector and the memory vector for thinking the LSTM layer to generate a final implicit strata state vector and a final memory vector for thinking the LSTM layer, and determining the final implicit strata state vector as a reasoning word vector, and finally, performing linear transformation on the reasoning word vector, and combining with an activation function to generate a prediction result. Therefore, the remote sensing image is subjected to a description algorithm based on feature enhancement and multi-step thinking attention, so that the accuracy of a remote sensing image prediction result is improved.

Description

Technical Field

[0001] The present application belongs to the field of computer application technology, and in particular, relates to a remote sensing image prediction method, device, terminal equipment and storage medium. Background Art

[0002] As a joint technology of image understanding and language generation, high-resolution remote sensing image description has attracted the attention of more and more researchers. Specifically, the high-resolution remote sensing image description task can generate a comprehensive and appropriate natural language based on the content of the image. Due to the novelty and creativity of this technology, high-resolution remote sensing image description has certain application prospects in civil and military fields, including human-computer interaction, blind assistants, battlefield environment analysis, etc.

[0003] In the related technology, the current remote sensing image prediction algorithm has the problem of poor accuracy. Summary of the invention

[0004] The embodiments of the present application provide a remote sensing image prediction method, apparatus, terminal device, and storage medium, which can solve the problem of poor accuracy of current remote sensing image prediction algorithms.

[0005] In the first aspect, an embodiment of the present application provides a remote sensing image prediction method, including: obtaining a primary feature vector corresponding to the remote sensing image according to a preset convolutional neural network model; performing feature enhancement on the primary feature vector according to a multi-head attention mechanism to generate an enhanced feature vector; generating an nth hidden state vector and a memory vector of a thinking LSTM layer according to the nth attention result, the nth query vector, and the n-1th hidden state vector and memory vector of the thinking LSTM layer, wherein the nth query vector is generated by linearly transforming the hidden state vector of the first LSTM layer and the n-1th hidden state vector of the thinking LSTM layer, and the nth attention result is generated by combining the nth query vector and the enhanced feature vector with a spatial attention mechanism, where n is the number of thoughts of the thinking attention mechanism and n is an integer greater than or equal to 0; substituting the optimal number of thoughts into the nth hidden state vector and memory vector of the thinking LSTM layer to generate a final hidden state vector and a final memory vector of the thinking LSTM layer, and determining the final hidden state of the thinking LSTM layer as an inference word vector; performing a linear transformation on the inference word vector and combining it with a softmax activation function to generate a prediction result.

[0006] In a possible implementation manner of the first aspect, the step of performing feature enhancement on the primary feature vector according to the multi-head attention mechanism to generate an enhanced feature vector includes:

[0007] Perform three independent linear mappings on the primary feature vector to obtain the query vector, key vector, and value vector corresponding to the primary feature vector respectively;

[0008] Combine the query vector, key vector, and value vector corresponding to the primary feature vector with the multi-head attention mechanism to generate a transition feature vector;

[0009] Performing linear transformation on the query vector and transition feature vector corresponding to the primary feature vector to generate a vector to be filtered;

[0010] The query vector and transition feature vector corresponding to the primary feature vector are linearly transformed and combined with the sigmoid function to generate a filter gate;

[0011] The filter gate is applied to the vector to be filtered according to element-by-element multiplication to generate an enhanced feature vector.

[0012] Optionally, in another possible implementation of the first aspect, before substituting the optimal number of thoughts into the nth hidden state vector and memory vector of the thinking LSTM layer, generating the final hidden state vector and the final memory vector of the thinking LSTM layer, and determining the final hidden state of the thinking LSTM layer as the inference word vector, the method further includes:

[0013] When n is equal to 0, the hidden state vector of the first LSTM layer is combined with the sigmoid function to generate a first confidence, and the optimal number of reflections is determined according to the first confidence and a preset threshold, or;

[0014] When n is greater than 0, the nth hidden state vector of the LSTM layer is considered and combined with the sigmoid function to generate a first confidence level, and the optimal number of considerations is determined based on the first confidence level and a preset threshold.

[0015] Optionally, in another possible implementation of the first aspect, substituting the optimal number of thoughts into the n-th hidden state vector and memory vector of the thinking LSTM layer to generate the final hidden state vector and final memory vector of the thinking LSTM layer includes:

[0016] Determine a second confidence level according to the first confidence level, and normalize the second confidence level;

[0017] Determine the final hidden state vector of the thinking LSTM layer according to the second confidence, the hidden state vector of the first LSTM layer, the optimal number of reflections, and the nth hidden state vector of the thinking LSTM layer;

[0018] According to the second confidence, the memory vector of the first LSTM layer, the optimal number of thinking times and the nth memory vector of the thinking LSTM layer, the final memory vector of the thinking LSTM layer is determined.

[0019] Optionally, in another possible implementation manner of the first aspect, after the linear transformation of the inference word vector at the current moment is combined with a softmax activation function to generate a prediction result, the method further includes:

[0020] Determine the loss function corresponding to the number of thoughts of the attention mechanism according to the preset hyperparameters, the number of thoughts of the attention mechanism, and the first confidence level;

[0021] According to the loss value corresponding to the loss function, optimize the number of thoughts of the attention mechanism.

[0022] In a second aspect, an embodiment of the present application provides a remote sensing image prediction device, comprising: an acquisition module, used to acquire a primary feature vector corresponding to the remote sensing image according to a preset convolutional neural network model; a first generation module, used to perform feature enhancement on the primary feature vector according to a multi-head attention mechanism to generate an enhanced feature vector; a second generation module, used to generate an nth hidden state vector and a memory vector of a thinking LSTM layer according to the nth attention result, the nth query vector, and the n-1th hidden state vector and memory vector of the thinking LSTM layer, wherein the nth query vector is generated based on the hidden state vector of the first LSTM layer and the n-1th hidden state vector of the thinking LSTM layer. The nth hidden state vector is generated by linear transformation, and the nth attention result is generated by the nth query vector and the enhanced feature vector in combination with the spatial attention mechanism, where n is the number of thoughts of the thinking attention mechanism and n is an integer greater than or equal to 0; the first determination module is used to substitute the optimal number of thoughts into the nth hidden state vector and memory vector of the thinking LSTM layer, generate the final hidden state vector and final memory vector of the thinking LSTM layer, and determine the final hidden state of the thinking LSTM layer as the reasoning word vector; the third generation module is used to generate a prediction result after linear transformation of the reasoning word vector in combination with the softmax activation function.

[0023] In a possible implementation manner of the second aspect, the first generating module includes:

[0024] An acquisition unit, used for performing three independent linear mappings on the primary feature vector to respectively acquire a query vector, a key vector, and a value vector corresponding to the primary feature vector;

[0025] A first generating unit, configured to generate a transition feature vector by combining a query vector, a key vector, and a value vector corresponding to the primary feature vector with a multi-head attention mechanism;

[0026] A second generating unit is used to perform linear transformation on the query vector and the transition feature vector corresponding to the primary feature vector to generate a vector to be filtered;

[0027] The third generating unit is used to perform linear transformation on the query vector and the transition feature vector corresponding to the primary feature vector and then combine them with the sigmoid function to generate a filtering gate;

[0028] The fourth generating unit is used to apply the filtering gate to the vector to be filtered according to element-by-element multiplication to generate an enhanced feature vector.

[0029] Optionally, in another possible implementation manner of the second aspect, the apparatus further includes:

[0030] A second determination module is used for combining the hidden state vector of the first LSTM layer with the sigmoid function to generate a first confidence when n is equal to 0, and determining the optimal number of reflections according to the first confidence and a preset threshold;

[0031] The third determination module is used to combine the n-th hidden state vector of the LSTM layer with the sigmoid function when n is greater than 0 to generate a first confidence level, and determine the optimal number of reflections based on the first confidence level and a preset threshold.

[0032] Optionally, in another possible implementation manner of the second aspect, the first determining module includes:

[0033] A first determining unit, configured to determine a second confidence level according to the first confidence level;

[0034] A normalization unit, used for normalizing the second confidence;

[0035] A second determining unit is used to determine a final hidden state vector of the thinking LSTM layer according to the second confidence, the hidden state vector of the first LSTM layer, the optimal number of reflections, and the nth hidden state vector of the thinking LSTM layer;

[0036] The third determining unit is used to determine the final memory vector of the thinking LSTM layer according to the second confidence, the memory vector of the first LSTM layer, the optimal number of thinking times, and the nth memory vector of the thinking LSTM layer.

[0037] Optionally, in another possible implementation manner of the second aspect, the apparatus further includes:

[0038] A fourth determination module is used to determine a loss function corresponding to the number of thoughts of the attention mechanism according to a preset hyperparameter, the number of thoughts of the attention mechanism, and the first confidence level;

[0039] The optimization module is used to optimize the number of thoughts of the attention mechanism according to the loss value corresponding to the loss function.

[0040] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the remote sensing image prediction method as described above when executing the computer program.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for detecting respiratory frequency as described above is implemented.

[0042] A remote sensing image prediction method provided in an embodiment of the present application first obtains a primary feature vector corresponding to the remote sensing image through a preset convolutional neural network model, and then combines the primary feature vector with a description algorithm based on feature enhancement and multi-step thinking attention, thereby improving the accuracy of the remote sensing image prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 It is a flowchart of a remote sensing image prediction method provided by an embodiment of the present application;

[0045] Figure 2 It is a structural diagram of a design model provided by an embodiment of the present application;

[0046] Figure 3 It is a flow chart of an enhanced visual feature module for visual domain interaction provided by an embodiment of the present application;

[0047] Figure 4 is a flow chart of a multi-step thinking attention mechanism provided by an embodiment of the present application;

[0048] Figure 5 It is a schematic diagram of the structure of a remote sensing image prediction device provided in one embodiment of the present application;

[0049] Figure 6 It is a structural diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0051] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0052] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0053] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0054] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0055] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0056] The remote sensing image prediction method, apparatus, terminal device and storage medium provided in the present application are described in detail below with reference to the accompanying drawings.

[0057] Figure 1 A schematic flow chart of a remote sensing image prediction method provided in an embodiment of the present application is shown.

[0058] Step 101, obtaining a primary feature vector corresponding to the remote sensing image according to a preset convolutional neural network model.

[0059] Preferably, the preset convolutional neural network model can be a VGG16 model. The VGG16 model is a deep network constructed by stacking multiple smaller convolutional layers and pooling layers to enhance the expression ability of the model. Specifically, the VGG16 model consists of 16 convolutional layers and 3 fully connected layers. Among them, the convolutional layer is mainly used to extract the features of the input image, while the fully connected layer is used to map the extracted features to the category probability. By using the VGG16 model as the preset convolutional neural network model, it is helpful to capture the complex features of remote sensing images while reducing the number of parameters.

[0060] In one embodiment, the VGG16 model is used to extract the primary feature vector corresponding to the remote sensing image, and the primary feature vector is the visual feature corresponding to the remote sensing image (that is, the feature information output by the last layer of the VGG16 model, marked as F).

[0061] Step 102: perform feature enhancement on the primary feature vector according to a multi-head attention mechanism to generate an enhanced feature vector.

[0062] Among them, the multi-head attention mechanism can refer to an attention mechanism commonly used in deep learning, especially in the Transformer model. Its core idea is to process input information in parallel through multiple different attention "heads", so that the model can understand the input features from different angles and scales.

[0063] In an embodiment of the present application, the primary feature vector is feature enhanced through a multi-head attention mechanism to generate an enhanced feature vector, thereby increasing the model's ability to recognize and apply key features.

[0064] Furthermore, in a possible implementation of the embodiment of the present application, the above step 102 may include:

[0065] Step 1021, perform three independent linear mappings on the primary feature vector to respectively obtain a query vector, a key vector, and a value vector corresponding to the primary feature vector.

[0066] Step 1022, the query vector, key vector and value vector corresponding to the primary feature vector are combined with a multi-head attention mechanism to generate a transition feature vector.

[0067] Step 1023: linearly transform the query vector and the transition feature vector corresponding to the primary feature vector to generate a vector to be filtered.

[0068] Step 1024, linearly transform the query vector and transition feature vector corresponding to the primary feature vector and combine them with the sigmoid function to generate a filter gate.

[0069] Step 1025 , applying the filter gate to the vector to be filtered according to element-by-element multiplication to generate an enhanced feature vector.

[0070] According to the above steps 1021-1025, in one embodiment of the present application, the multi-head attention mechanism first performs three independent linear mappings on the primary feature vector F to obtain the query vector Q, key vector K and value vector V corresponding to the primary feature vector F, namely:

[0071] Q=FW Q ; K = FW K ; V = FW V

[0072] Among them, W Q , W K and W V They are the learnable weight matrices corresponding to the query vector Q, key vector K, and value vector V respectively.

[0073] Then, according to the multi-head attention mechanism, the similarity between the query vector Q and the key vector K is calculated to process the mutual interaction of the features between the grids and form the final grid interaction visual feature, that is, the transition feature vector F′:

[0074] F′=f MHA (Q,K,V)

[0075] Then, two parallel linear transformations with non-shared parameters are used to generate the vector to be filtered and the filter gate respectively, thereby achieving the desired filtering function. The calculation process is as follows:

[0076]

[0077] Among them, x is the vector to be filtered, g is the filtering gate, Q is the query vector, F′ is the transition feature vector, and σ is the activation function used to control the output range from 0 to 1. and are the weight matrices applied to Q and F′, respectively, and b x is the offset, and are also the weight matrices applied to Q and F′, respectively, and b g is another offset.

[0078] Finally, the filter gate is applied to the vector to be filtered according to element-by-element multiplication to generate an enhanced feature vector, which is the final feature-enhanced visual feature, namely:

[0079] Z = g·x

[0080] Among them, Z is the enhanced feature vector, g is the filtering gate, and x is the vector to be filtered.

[0081] Step 103, based on the nth attention result, the nth query vector, and the n-1th hidden state vector and memory vector of the thinking LSTM layer, generate the nth hidden state vector and memory vector of the thinking LSTM layer. Among them, the nth query vector is generated by linear transformation based on the hidden state vector of the first LSTM layer and the n-1th hidden state vector of the thinking LSTM layer, the nth attention result is generated by combining the nth query vector and the enhanced feature vector with the spatial attention mechanism, and n is the number of thoughts of the thinking attention mechanism and n is an integer greater than or equal to 0.

[0082] In the embodiment of the present application, it is first necessary to construct the nth query vector q corresponding to the attention mechanism t,n , the calculation formula is as follows:

[0083]

[0084] in, is the hidden state vector of the first LSTM layer, To consider the n-1th hidden state vector of the LSTM layer, W q is the weight matrix, b q is the offset.

[0085] Next, q t,n As the query vector of the recurrent attention mechanism, it aims to obtain the nth attention result The calculation formula is as follows:

[0086]

[0087] Among them, f att is the spatial attention mechanism, and Z is the enhanced feature vector.

[0088] Finally, according to the nth attention result, the nth query vector, and the n-1th hidden state vector and memory vector of the thinking LSTM layer, the nth hidden state vector and memory vector of the thinking LSTM layer are generated. The formula is as follows:

[0089]

[0090] in, To think about the nth hidden state vector of the LSTM layer, To think about the nth memory vector of the LSTM layer, is the nth attention result, is the nth query vector of the first LSTM layer, To consider the n-1th hidden state vector of the LSTM layer, Consider the n-1th memory vector of the LSTM layer.

[0091] Step 104, substitute the optimal number of thinking times into the nth hidden state vector and memory vector of the thinking LSTM layer, generate the final hidden state vector and final memory vector of the thinking LSTM layer, and determine the final hidden state of the thinking LSTM layer as the inference word vector.

[0092] As a possible implementation method, the above optimal number of thinking times can be determined according to the following steps:

[0093] When n is equal to 0, the hidden state vector of the first LSTM layer is combined with the sigmoid function to generate a first confidence, and the optimal number of reflections is determined according to the first confidence and a preset threshold, or;

[0094] When n is greater than 0, the nth hidden state vector of the LSTM layer is considered and combined with the sigmoid function to generate a first confidence level, and the optimal number of considerations is determined based on the first confidence level and a preset threshold.

[0095] It should be noted that the first confidence level can be classified according to the number of thoughts, and the formula is as follows:

[0096]

[0097] Among them, p t,n is the first confidence, σ is the activation function, which is used to control the output range from 0 to 1. is the hidden state vector of the first LSTM layer, To think about the nth hidden state vector of the LSTM layer, W1 and W2 are weight matrices, and b1 and b2 are biases.

[0098] After determining the first confidence level, the optimal number of reflections N(t) is determined based on the first confidence level and the preset threshold. The formula is as follows:

[0099]

[0100] Among them, p t,n is the first confidence level, ε is the preset threshold, which is a small value greater than 0. Preferably, it can be set to 1×10 -4 As another possible implementation, the optimal number of reflections N(t) may also be determined according to a preset maximum number of cycles M, a first confidence level, and a preset threshold value, as shown in the following formula:

[0101]

[0102] In the embodiment of the present application, using confidence to dynamically determine the number of thoughts of the attention mechanism is a key step. The core purpose of this mechanism is to decide whether to continue the cyclic attention operation based on the model's confidence in the current state, thereby avoiding unnecessary calculations and improving efficiency.

[0103] In an embodiment of the present application, as a possible implementation manner, the above step 104 may also include:

[0104] Step 1041, determining a second confidence level according to the first confidence level, and normalizing the second confidence level.

[0105] The formula corresponding to the above step 1041 is as follows:

[0106]

[0107] Among them, β t,n is the second confidence level, p t,n is the first confidence level corresponding to when n is greater than 0, p t,0 It is the first confidence level corresponding to when n is equal to 0.

[0108] In order to make the sum of the weights of the second confidence equal to 1, the second confidence can be normalized as follows:

[0109]

[0110] Step 1042, determining the final hidden state vector of the thinking LSTM layer according to the second confidence, the hidden state vector of the first LSTM layer, the optimal number of reflections, and the nth hidden state vector of the thinking LSTM layer.

[0111] Step 1043, determining the final memory vector of the thinking LSTM layer according to the second confidence, the memory vector of the first LSTM layer, the optimal number of thinking times, and the nth memory vector of the thinking LSTM layer.

[0112] In one embodiment, the contents of steps 1042-1043 above can be expressed by the following formula:

[0113]

[0114] Among them, N(t) is the optimal number of thinking times, To think about the final hidden state vector of the LSTM layer, To think about the final memory vector of the LSTM layer, is the hidden state vector of the first LSTM layer, is the memory vector of the first LSTM layer, To think about the nth hidden state vector of the LSTM layer, Consider the n-th memory vector of the LSTM layer.

[0115] In the embodiment of the present application, the final hidden layer state vector needs to be determined as the inference word vector, that is:

[0116]

[0117] Among them, c t is the inference word vector, Consider the final hidden state vector of the LSTM layer.

[0118] Step 105, perform linear transformation on the inference word vector and combine it with the softmax activation function to generate a prediction result.

[0119] In the present embodiment, c t Through a linear connection layer with softmax function, the prediction result p of the inference word vector is realized t :

[0120] p t =softmax(W p c t +b p )

[0121] Among them, c t is the inference word vector, W p is the weight matrix, b p is the offset.

[0122] In an embodiment of the present application, as a possible implementation manner, after the above step 105, the following is further included:

[0123] Determine the loss function corresponding to the number of thoughts of the attention mechanism according to the preset hyperparameters, the number of thoughts of the attention mechanism, and the first confidence level;

[0124] According to the loss value corresponding to the loss function, optimize the number of thoughts of the attention mechanism.

[0125] It should be noted that for the multi-step thinking attention mechanism, a loss function with a thinking number constraint is added to adapt to the reasoning process of the decoder and maximize the guidance of effective information to the decoder. The formula is defined as follows:

[0126]

[0127] in, is the loss function corresponding to the number of thoughts, λatt is a preset hyperparameter, preferably, it can be set to 1×10 -4 , in this loss function, (n+1)(1-p t,n ) can promote p t,n The value of is as large as possible to reduce the number of times of thinking.

[0128] A remote sensing image prediction method provided in an embodiment of the present application first obtains a primary feature vector corresponding to the remote sensing image through a preset convolutional neural network model, and then combines the primary feature vector with a description algorithm based on feature enhancement and multi-step thinking attention, thereby improving the accuracy of the remote sensing image prediction results.

[0129] like Figure 2 The structural diagram of the design model shown in FIG. 1 is a diagram of a remote sensing image prediction method provided in an embodiment of the present application, and can be applied to the above model. Figure 2 As shown in the figure, the remote sensing image is firstly extracted according to the encoder, and then the feature is enhanced according to the multi-head attention mechanism. Then the enhanced feature vector is input into the first LSTM layer and the thinking LSTM layer in turn, and finally combined with the softmax function, the prediction result corresponding to the remote sensing image is input.

[0130] like Figure 3 The flowchart of the enhanced visual feature module for visual domain interaction is shown in detail. Figure 2 The process of feature enhancement step in .

[0131] like Figure 4 The flowchart of the multi-step thinking attention mechanism shown in detail Figure 2 The enhanced feature vector is input into the first LSTM layer and the thinking LSTM layer in turn.

[0132] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0133] Corresponding to the prediction method of remote sensing images in the above embodiment, Figure 5 A structural block diagram of a remote sensing image prediction device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0134] Reference Figure 5 , the device 500 comprises:

[0135] An acquisition module 501 is used to acquire a primary feature vector corresponding to a remote sensing image according to a preset convolutional neural network model;

[0136] A first generating module 502 is used to perform feature enhancement on the primary feature vector according to the multi-head attention mechanism to generate an enhanced feature vector;

[0137] The second generation module 503 is used to generate the nth hidden state vector and memory vector of the thinking LSTM layer according to the nth attention result, the nth query vector, and the n-1th hidden state vector and memory vector of the thinking LSTM layer, wherein the nth query vector is generated by linearly transforming the hidden state vector of the first LSTM layer and the n-1th hidden state vector of the thinking LSTM layer, and the nth attention result is generated by combining the nth query vector and the enhanced feature vector with the spatial attention mechanism, and n is the number of thoughts of the thinking attention mechanism and n is an integer greater than or equal to 0;

[0138] The first determination module 504 is used to substitute the optimal number of thinking times into the nth hidden state vector and memory vector of the thinking LSTM layer, generate the final hidden state vector and final memory vector of the thinking LSTM layer, and determine the final hidden state of the thinking LSTM layer as the inference word vector;

[0139] The third generation module 505 is used to perform linear transformation on the inference word vector and combine it with the softmax activation function to generate a prediction result.

[0140] In actual use, the remote sensing image prediction device provided in the embodiment of the present application can be configured in any terminal device to execute the aforementioned remote sensing image prediction method.

[0141] The remote sensing image prediction device provided in the present application first obtains the primary feature vector corresponding to the remote sensing image through a preset convolutional neural network model, and then combines the primary feature vector with a description algorithm based on feature enhancement and multi-step thinking attention, thereby improving the accuracy of the remote sensing image prediction results.

[0142] In a possible implementation of the embodiment of the present application, the first generating module 502 includes:

[0143] An acquisition unit, used for performing three independent linear mappings on the primary feature vector to respectively acquire a query vector, a key vector, and a value vector corresponding to the primary feature vector;

[0144] A first generating unit, configured to generate a transition feature vector by combining a query vector, a key vector, and a value vector corresponding to the primary feature vector with a multi-head attention mechanism;

[0145] A second generating unit is used to perform linear transformation on the query vector and the transition feature vector corresponding to the primary feature vector to generate a vector to be filtered;

[0146] The third generating unit is used to perform linear transformation on the query vector and the transition feature vector corresponding to the primary feature vector and then combine them with the sigmoid function to generate a filtering gate;

[0147] The fourth generating unit is used to apply the filtering gate to the vector to be filtered according to element-by-element multiplication to generate an enhanced feature vector.

[0148] Furthermore, in another possible implementation of the embodiment of the present application, the apparatus 500 further includes:

[0149] A second determination module is used for combining the hidden state vector of the first LSTM layer with the sigmoid function to generate a first confidence when n is equal to 0, and determining the optimal number of reflections according to the first confidence and a preset threshold;

[0150] The third determination module is used to combine the n-th hidden state vector of the LSTM layer with the sigmoid function when n is greater than 0 to generate a first confidence level, and determine the optimal number of reflections based on the first confidence level and a preset threshold.

[0151] Furthermore, in another possible implementation of the embodiment of the present application, the first determining module 504 includes:

[0152] A first determining unit, configured to determine a second confidence level according to the first confidence level;

[0153] A normalization unit, used for normalizing the second confidence;

[0154] A second determining unit is used to determine a final hidden state vector of the thinking LSTM layer according to the second confidence, the hidden state vector of the first LSTM layer, the optimal number of reflections, and the nth hidden state vector of the thinking LSTM layer;

[0155] The third determining unit is used to determine the final memory vector of the thinking LSTM layer according to the second confidence, the memory vector of the first LSTM layer, the optimal number of thinking times, and the nth memory vector of the thinking LSTM layer.

[0156] Furthermore, in another possible implementation of the embodiment of the present application, the apparatus 500 further includes:

[0157] A fourth determination module is used to determine a loss function corresponding to the number of thoughts of the attention mechanism according to a preset hyperparameter, the number of thoughts of the attention mechanism, and the first confidence level;

[0158] The optimization module is used to optimize the number of thoughts of the attention mechanism according to the loss value corresponding to the loss function.

[0159] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0160] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0161] In order to implement the above embodiments, the present application also proposes a terminal device.

[0162] Figure 6 A schematic diagram of the structure of a terminal device according to an embodiment of the present application.

[0163] like Figure 6 As shown, the terminal device 200 includes:

[0164] The memory 210 and at least one processor 220, a bus 230 connecting different components (including the memory 210 and the processor 220), the memory 210 stores a computer program, and when the processor 220 executes the program, the remote sensing image prediction method described in the embodiment of the present application is implemented.

[0165] Bus 230 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0166] The terminal device 200 typically includes a variety of electronic device readable media, which can be any available media that can be accessed by the terminal device 200, including volatile and non-volatile media, removable and non-removable media.

[0167] The memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. The terminal device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 260 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to bus 230 via one or more data medium interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0168] A program / utility 280 having a set (at least one) of program modules 270 may be stored, for example, in the memory 210, such program modules 270 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 270 generally perform the functions and / or methods of the embodiments described herein.

[0169] The terminal device 200 can also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), and can also communicate with one or more devices that enable users to interact with the terminal device 200, and / or communicate with any device that enables the terminal device 200 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be carried out through an input / output (I / O) interface 292. In addition, the terminal device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 293. As shown in the figure, the network adapter 293 communicates with other modules of the terminal device 200 through a bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the terminal device 200, including but not limited to: microcode, device driver, redundant processing unit, external disk drive array, RAID system, tape drive, and data backup storage system, etc.

[0170] The processor 220 executes various functional applications and data processing by running the programs stored in the memory 210 .

[0171] It should be noted that the implementation process and technical principles of the terminal device of this embodiment refer to the aforementioned explanation of the remote sensing image prediction method of the embodiment of the present application, and will not be repeated here.

[0172] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0173] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0174] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0175] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0176] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0177] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A remote sensing image prediction method, characterized in that: include: Obtaining the primary feature vector corresponding to the remote sensing image according to the preset convolutional neural network model; Performing feature enhancement on the primary feature vector according to a multi-head attention mechanism to generate an enhanced feature vector; Generate the nth hidden state vector and memory vector of the thinking LSTM layer according to the nth attention result, the nth query vector, and the n-1th hidden state vector and memory vector of the thinking LSTM layer, wherein the nth query vector is generated by linearly transforming the hidden state vector of the first LSTM layer and the n-1th hidden state vector of the thinking LSTM layer, and the nth attention result is generated by combining the nth query vector and the enhanced feature vector with the spatial attention mechanism, where n is the number of thoughts of the thinking attention mechanism and n is an integer greater than or equal to 0; Substituting the optimal number of thoughts into the nth hidden state vector and memory vector of the thinking LSTM layer, generating a final hidden state vector and a final memory vector of the thinking LSTM layer, and determining the final hidden state of the thinking LSTM layer as the inference word vector; The inference word vector is linearly transformed and combined with a softmax activation function to generate a prediction result.

2. The method according to claim 1, characterized in that The step of performing feature enhancement on the primary feature vector according to the multi-head attention mechanism to generate an enhanced feature vector includes: Performing three independent linear mappings on the primary feature vector to respectively obtain a query vector, a key vector, and a value vector corresponding to the primary feature vector; The query vector, key vector and value vector corresponding to the primary feature vector are combined with a multi-head attention mechanism to generate a transition feature vector; Performing linear transformation on the query vector corresponding to the primary feature vector and the transition feature vector to generate a vector to be filtered; The query vector corresponding to the primary feature vector and the transition feature vector are linearly transformed and combined with a sigmoid function to generate a filter gate; The filter gate is applied to the vector to be filtered according to element-by-element multiplication to generate the enhanced feature vector.

3. The method according to claim 1, characterized in that Before substituting the optimal number of thoughts into the nth hidden state vector and memory vector of the thinking LSTM layer to generate the final hidden state vector and final memory vector of the thinking LSTM layer, and determining the final hidden state of the thinking LSTM layer as the inference word vector, the method further includes: When n is equal to 0, the hidden state vector of the first LSTM layer is combined with a sigmoid function to generate a first confidence, and the optimal number of reflections is determined according to the first confidence and a preset threshold, or; When n is greater than 0, the nth hidden layer state vector of the thinking LSTM layer is combined with a sigmoid function to generate the first confidence, and the optimal number of thinking times is determined according to the first confidence and the preset threshold.

4. The method according to claim 3, characterized in that Substituting the optimal number of thoughts into the nth hidden state vector and memory vector of the thinking LSTM layer to generate the final hidden state vector and final memory vector of the thinking LSTM layer includes: Determine a second confidence level according to the first confidence level, and normalize the second confidence level; Determine a final hidden state vector of the thinking LSTM layer according to the second confidence, the hidden state vector of the first LSTM layer, the optimal number of reflections, and the nth hidden state vector of the thinking LSTM layer; The final memory vector of the thinking LSTM layer is determined according to the second confidence, the memory vector of the first LSTM layer, the optimal number of reflections, and the nth memory vector of the thinking LSTM layer.

5. The method according to any one of claims 1 to 4, characterized in that: After the linear transformation of the inference word vector at the current moment is combined with a softmax activation function to generate a prediction result, the method further includes: Determine a loss function corresponding to the number of thoughts of the thinking attention mechanism according to a preset hyperparameter, the number of thoughts of the thinking attention mechanism, and the first confidence level; According to the loss value corresponding to the loss function, the number of thoughts of the thinking attention mechanism is optimized.

6. A remote sensing image prediction device, characterized in that: include: An acquisition module, used for acquiring a primary feature vector corresponding to a remote sensing image according to a preset convolutional neural network model; A first generating module is used to perform feature enhancement on the primary feature vector according to a multi-head attention mechanism to generate an enhanced feature vector; The second generation module is used to generate the nth hidden state vector and memory vector of the thinking LSTM layer according to the nth attention result, the nth query vector, and the n-1th hidden state vector and memory vector of the thinking LSTM layer, wherein the nth query vector is generated by linearly transforming the hidden state vector of the first LSTM layer and the n-1th hidden state vector of the thinking LSTM layer, and the nth attention result is generated by combining the nth query vector and the enhanced feature vector with the spatial attention mechanism, and n is the number of thoughts of the thinking attention mechanism and n is an integer greater than or equal to 0; A first determination module is used to substitute the optimal number of thoughts into the nth hidden state vector and memory vector of the thinking LSTM layer, generate a final hidden state vector and a final memory vector of the thinking LSTM layer, and determine the final hidden state of the thinking LSTM layer as an inference word vector; The third generation module is used to perform linear transformation on the inference word vector and combine it with a softmax activation function to generate a prediction result.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images