A small sample image matching method based on multi-generalization knowledge accumulation

Abstract

The present invention discloses a small sample image matching method based on multi-generalized knowledge accumulation, comprising: acquiring a first optical image and a first SAR image; preprocessing the first optical image and the first SAR image to obtain multiple first optical image slices and multiple first SAR image slices; inputting the multiple first optical image slices into a first branch of a matching model and inputting the multiple first SAR image slices into a second branch of the matching model to obtain a first eigenvector and a second eigenvector; wherein the matching model is a two-branch network trained based on a small sample subtask, a cross-scene subtask, a first support set of the small sample subtask, a first query set, and a second support set and a second query set of the cross-scene subtask; and obtaining a matching result based on the similarity between the first eigenvector and the second eigenvector. The generalized knowledge learning strategy adopted by the present invention enables the matching model to be well applicable to small sample situations in different scenarios, thereby improving the versatility of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a small sample image matching method based on multi-generalization knowledge accumulation. Background Art

[0002] With the rapid development of remote sensing technology, sensor types are becoming increasingly diverse, resulting in the acquisition of massive amounts of heterogeneous remote sensing images. Heterogeneous image registration technology uses image matching algorithms to establish spatial correspondences between heterogeneous images. Based on these correspondences, the images are transformed to align their positional information. This lays a solid foundation for the integrated utilization of the rich remote sensing images acquired by different sensors for practical applications. As a core step in image registration, the accuracy of image matching algorithms directly impacts its accuracy.

[0003] At present, with the rapid development of deep learning, the method of applying deep learning to heterogeneous remote sensing image matching has received widespread attention. However, in the application scenarios of remote sensing image matching tasks, it is difficult to obtain data for some scenes. Therefore, how to solve the problem of small data samples is very critical.

[0004] However, existing technologies haven't addressed this practical issue. For one thing, when existing image matching techniques are applied to small sample sizes for specific scenarios, they either directly use small sample data for network learning, or train the matching network using a large amount of data from other scenarios before fine-tuning the pre-trained network based on the small sample data from the specific scenario. This approach to learning deep networks based solely on small sample data can lead to overfitting, resulting in reduced network matching performance. Furthermore, while meta-learning, as a typical generalized knowledge learning strategy, can effectively address the small sample size issue, when applied to small sample size matching for specific scenarios, if the data distribution for other scenarios differs significantly from that for the specific scenario, the small sample size generalized knowledge accumulated by fitting the distribution of small sample data from other scenarios may not be applicable to the specific scenario. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a small sample image matching method based on multi-generalization knowledge accumulation. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a small sample image matching method based on multi-generalization knowledge accumulation, comprising:

[0007] Acquiring a first optical image and a first SAR image to be matched;

[0008] Preprocessing the first optical image and the first SAR image to obtain a plurality of first optical image slices and a plurality of first SAR image slices;

[0009] The first optical image slices are input into the first branch of the matching model, and the first SAR image slices are input into the second branch of the matching model to obtain the first eigenvectors e o and the second eigenvector e s ; Wherein, the matching model is: a two-branch network trained based on the Few Shot small sample subtask, the cross-scene subtask, the first support set of the small sample subtask, the first query set, and the second support set and second query set of the cross-scene subtask;

[0010] According to the first eigenvector e o With the second eigenvector e s The similarity between the optical image and the SAR image is obtained.

[0011] In one embodiment of the present invention, the matching model is trained according to the following steps:

[0012] Acquire a plurality of optical-SAR image pairs under different scenes from a data set, each of the optical-SAR image pairs comprising a second optical image and a corresponding second SAR image;

[0013] Each optical-SAR image pair is preprocessed to form a training data set P = {p1, p2, ..., p N}, where p n represents training data of the nth scene, where n=1, 2, ..., N, and the training data of each scene includes a plurality of second optical image slices of the scene and a plurality of corresponding second SAR image slices;

[0014] Constructing a two-branch network to be trained and initializing parameters; the two-branch network to be trained includes a first branch and a second branch, wherein the first branch and the second branch each include seven convolutional layers;

[0015] Taking any one of the different scenarios as a specific scenario and the remaining scenarios as auxiliary scenarios, and constructing a data set for the i-th small sample subtask and a data set for the i-th cross-scenario subtask based on the training data of the specific scenario and the training data of the auxiliary scenarios respectively;

[0016] Based on the data set of the i-th small sample subtask and the data set of the i-th cross-scene subtask, the two-branch network to be trained is trained alternately until the accuracy of the two-branch network reaches a preset condition, and the trained two-branch network is determined as the matching model.

[0017] In one embodiment of the present invention, each optical-SAR image pair is preprocessed to form a training data set P under different scenarios = {p1, p2, ..., p N} steps, including:

[0018] For each optical-SAR image pair, after extracting feature points in the second optical image using the SIFT algorithm, multiple second optical image slices are obtained by cropping with each feature point as the center, and multiple second SAR image slices are obtained by cropping at the same position of the second SAR image corresponding to the second optical image, forming the training data set P under different scenarios = {p1, p2, ..., p N}, where p n Represents the training data of the nth scene.

[0019] In one embodiment of the present invention, any one of the different scenarios is used as a specific scenario and the remaining scenarios are used as auxiliary scenarios. Based on the training data of the specific scenario and the training data of the auxiliary scenarios, the steps of respectively constructing a data set for the i-th small sample subtask and a data set for the i-th cross-scenario subtask include:

[0020] Taking any one of the different scenarios as a specific scenario and the remaining scenarios as auxiliary scenarios;

[0021] Obtain the first amount of training data in any auxiliary scenario to form the first support set And obtain the second amount of training data in the auxiliary scenario to form the first query set The data set that constitutes the i-th small sample subtask wherein the first data amount is smaller than the second data amount;

[0022] Obtain the first amount of training data in any auxiliary scenario to form the second support set And obtain the second amount of training data in another auxiliary scenario to form a second query set The data set that constitutes the i-th cross-scenario subtask

[0023] In one embodiment of the present invention, the step of alternately training the to-be-trained dual-branch network based on the data set of the i-th small sample subtask and the data set of the i-th cross-scenario subtask until the accuracy of the dual-branch network reaches a preset condition, and determining the trained dual-branch network as the matching model includes:

[0024] The first support set The second optical image slice and the corresponding second SAR image slice in the image are input to the first branch and the second branch respectively, to obtain a first current feature vector output by the first branch and a second current feature vector output by the second branch;

[0025] Determine a first loss value based on the first current feature vector, the second current feature vector, and a ternary loss function, and update parameters of the dual-branch network according to the first loss value and a first preset step size;

[0026] Based on the updated parameter θ', the first query set The second optical image slice and the corresponding second SAR image slice in the image are input to the first branch and the second branch respectively, to obtain a third current eigenvector output by the first branch and a fourth current eigenvector output by the second branch;

[0027] Based on the third current eigenvector, the fourth current eigenvector and the ternary loss function, a second loss value is determined, and the parameters of the branch network are updated from θ' to θ according to the second loss value and the second preset step size. 1 ;

[0028] Based on the updated parameters θ 1 , the second support set The second optical image slice and the corresponding second SAR image in the image are input to the first branch and the second branch respectively, to obtain the fifth current eigenvector output by the first branch and the sixth current eigenvector output by the second branch;

[0029] Determining a third loss value based on the fifth current eigenvector, the sixth current eigenvector, and the ternary loss function, and updating parameters of the dual-branch network according to the third loss value and a first preset step size;

[0030] According to the updated parameters θ 2 , the second query set The second optical image slice and the corresponding second SAR image in the image are input to the first branch and the second branch respectively, to obtain the seventh current eigenvector output by the first branch and the eighth current eigenvector output by the second branch;

[0031] Determining a fourth loss value based on the fifth current eigenvector, the sixth current eigenvector, and the ternary loss function, and updating parameters of the dual-branch network according to the fourth loss value and a second preset step size;

[0032] Determine whether the fourth loss value meets the preset conditions; if so, obtain the trained dual-branch network; if not, set i = i + 1, return to obtain the training data of the first data volume in any auxiliary scenario, and form the first support set And obtain the second amount of training data in the auxiliary scenario to form the first query set The data set that constitutes the i-th small sample subtask steps.

[0033] In one embodiment of the present invention, based on the data set of the i-th small sample subtask and the data set of the i-th cross-scenario subtask, the two-branch network to be trained is alternately trained until the accuracy of the two-branch network reaches a preset condition, and after the step of determining the trained two-branch network as the matching model, the method further includes:

[0034] Optimize and adjust the parameters of the matching model.

[0035] In one embodiment of the present invention, the parameters of the matching model are optimized and adjusted according to the following formula:

[0036]

[0037] Where θ M represents the parameters of the matching model, D test training data representing a first amount of data for the specific scenario, represents the mapping function of the matching model, Represents the ternary loss function, and α represents the first preset step size.

[0038] In one embodiment of the present invention, according to the first eigenvector e o With the second eigenvector e s The step of obtaining a matching result between the optical image and the SAR image by comparing the similarity between the optical image and the SAR image comprises:

[0039] The first eigenvector e for each first optical image slice o , calculate the Euclidean distance between it and each first SAR image slice respectively;

[0040] The minimum value of the Euclidean distance is determined, and matching is performed with the first SAR image slice corresponding to the minimum value.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The matching model constructed in this invention can learn and accumulate effective small-sample generalization knowledge and cross-scenario generalization knowledge from other scenarios, which can be applied to small-sample image matching tasks in specific scenarios, improving matching performance. Compared to the existing technology that directly pre-trains the network using data from other scenarios and then fine-tunes it based on small-sample data from specific scenarios, the generalization knowledge learning strategy designed in this invention ultimately makes the matching model more suitable for small-sample situations in specific scenarios.

[0043] 2. The present invention helps the matching model learn cross-scenario generalization knowledge by constructing cross-scenario subtasks, which can reduce the impact of differences between different scene data on the matching results, and ensure that the final matching model can be well applied to small sample situations in different scenes, thereby improving the versatility of the model to a certain extent.

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of a small sample image matching method based on multi-generalized knowledge accumulation provided by an embodiment of the present invention;

[0046] Figure 2 Schematic diagram of the structure of a dual-branch network provided by an embodiment of the present invention;

[0047] Figure 3 3 is a schematic diagram of the training process of the matching model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0049] Figure 1 This is a flow chart of a small sample image matching method based on multi-generalized knowledge accumulation provided by an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a small sample image matching method based on multi-generalization knowledge accumulation, including:

[0050] S1. Acquire a first optical image and a first SAR image to be matched;

[0051] S2. Preprocessing the first optical image and the first SAR image to obtain a plurality of first optical image slices and a plurality of first SAR image slices;

[0052] S3, inputting a plurality of first optical image slices into the first branch of the matching model, inputting a plurality of first SAR image slices into the second branch of the matching model, and obtaining first eigenvectors e respectively. o and the second eigenvector e s ; Among them, the matching model is: a two-branch network trained based on the Few Shot small sample subtask, the cross-scene subtask, the first support set and the first query set of the small sample subtask, and the second support set and the second query set of the cross-scene subtask;

[0053] S4, according to the first eigenvector e o With the second eigenvector e sThe similarity between the two images is calculated to obtain the matching results between the optical image and the SAR image.

[0054] Figure 2 Schematic diagram of the structure of the dual-branch network provided by the embodiment of the present invention. Figure 2 As shown, in this embodiment, the matching model is trained according to the following steps:

[0055] Acquire multiple optical-SAR image pairs under different scenes from the dataset, each optical-SAR image pair including a second optical image and a corresponding second SAR image;

[0056] Each optical-SAR image pair is preprocessed to form a training data set P = {p1, p2, ..., p N}, where p n represents training data of the nth scene, where n=1, 2, ..., N, and the training data of each scene includes a plurality of second optical image slices of the scene and a plurality of corresponding second SAR image slices;

[0057] Constructing a two-branch network to be trained and initializing parameters; the two-branch network to be trained includes a first branch and a second branch, wherein the first branch and the second branch both include seven convolutional layers;

[0058] Take any one of the different scenarios as the specific scenario and the rest as auxiliary scenarios. Based on the training data of the specific scenario and the training data of the auxiliary scenarios, construct the data set of the i-th small sample subtask and the data set of the i-th cross-scenario subtask respectively.

[0059] Based on the data set of the i-th small sample subtask and the data set of the i-th cross-scene subtask, the two-branch network to be trained is trained alternately until the accuracy of the two-branch network reaches the preset conditions, and the trained two-branch network is determined as the matching model.

[0060] Optionally, feature points are extracted from optical images of different scenes in the optical-SAR image dataset SEN1-2, and optical-SAR image pairs with a number of feature points greater than a preset value are selected as training samples to ensure that the images used for training contain rich scene information, wherein the SEN1-2 dataset contains registered optical images and SAR images of different scenes.

[0061] In this embodiment, each optical-SAR image pair is preprocessed to form a training data set P = {p1, p2, ..., p N} steps, including:

[0062] For each optical-SAR image pair, after extracting feature points from the second optical image using the SIFT algorithm, multiple second optical image slices are obtained by cropping with each feature point as the center, and multiple second SAR image slices are obtained by cropping at the same position of the second SAR image corresponding to the second optical image, forming a training data set P = {p1, p2, ..., p N}, where p n Represents the training data of the nth scene.

[0063] Specifically, for the selected optical-SAR image, the SIFT algorithm is used to extract the feature points in the optical image, and multiple 64×64 optical image slices are obtained by cropping them with each feature point as the center. At the same time, SAR image slices that match the optical image slices are cropped at the corresponding positions of the registered SAR image to form the training data set P = {p1, p2, ..., p N}, where p n is the training data for the nth scene.

[0064] Figure 3 FIG. 1 is a schematic diagram of the training process of the matching model provided by an embodiment of the present invention. Figure 3 As shown, the dual-branch network to be trained includes a first branch and a second branch, both of which are a 7-layer deep feature extraction module. The deep feature extraction module includes the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer, the sixth convolutional layer and the seventh convolutional layer connected in sequence, wherein the convolution kernel size of the first to sixth convolutional layers is 3×3, the convolution kernel size of the seventh convolutional layer is 8×8, and the number of channels of the first to seventh convolutional layers are 32, 32, 64, 64, 128, 128, and 128 respectively. Batch normalization is used after each convolution layer. Further, two deep feature extraction modules with the same structure are arranged in parallel to form a dual-branch network to be trained, and then the parameters θ of the dual-branch network are initialized. 0 , and determine training parameters such as the first preset step size α, the second preset step size β, the number of training rounds epoches, etc.

[0065] In this embodiment, any one of the different scenarios is used as a specific scenario and the remaining scenarios are used as auxiliary scenarios. Based on the training data of the specific scenario and the training data of the auxiliary scenarios, the steps of respectively constructing a data set for the i-th small sample subtask and a data set for the i-th cross-scenario subtask include:

[0066] Take any one of the different scenes as a specific scene and the rest as auxiliary scenes;

[0067] Obtain the first amount of training data in any auxiliary scenario to form the first support set And obtain the second amount of training data in the auxiliary scenario to form the first query set The data set that constitutes the i-th small sample subtask wherein the first data amount is smaller than the second data amount;

[0068] Obtain the first amount of training data in any auxiliary scenario to form the second support set And obtain the second amount of training data in another auxiliary scenario to form a second query set The data set that constitutes the i-th cross-scenario subtask

[0069] During the training process, small sample subtasks and cross-scenario subtasks are first constructed. Specifically, the training data for a specific scenario is small sample data. Sample a data set with the same size as the small sample data of a specific scene from the training data of any auxiliary scene as the first support set of the small sample subtask At the same time, sufficient training data is sampled from the training data of the auxiliary scene as the first query set of the small sample subtask Constitute small sample subtasks Data collection Considering that the data distribution of auxiliary scenes and specific scenes may be quite different, this embodiment further constructs subtasks that can accumulate cross-scene generalization knowledge, namely cross-scene subtasks and cross-scene subtasks. The data set in is represented as Among them, a data set with the same size as the small sample data of the target scene is sampled from the training data of any auxiliary scene, and the second support set of the cross-scene subtask is Sample a sufficient data set from the training data of another auxiliary scene as the second query set for the cross-scene subtask

[0070] Exemplarily, based on the data set of the i-th small sample subtask and the data set of the i-th cross-scenario subtask, alternately training the dual-branch network to be trained until the accuracy of the dual-branch network reaches a preset condition, and then determining the trained dual-branch network as the matching model includes:

[0071] The first support set The second optical image slice and the corresponding second SAR image slice in the image are input to the first branch and the second branch respectively, to obtain a first current feature vector output by the first branch and a second current feature vector output by the second branch;

[0072] Determine a first loss value based on the first current eigenvector, the second current eigenvector, and the ternary loss function, and update parameters of the two-branch network according to the first loss value and a first preset step size;

[0073] Based on the updated parameter θ', the first query set The second optical image slice and the corresponding second SAR image slice in the image are input to the first branch and the second branch respectively, to obtain a third current eigenvector output by the first branch and a fourth current eigenvector output by the second branch;

[0074] Based on the third current eigenvector, the fourth current eigenvector and the ternary loss function, a second loss value is determined, and the parameters of the branch network are updated from θ' to θ according to the second loss value and the second preset step size. 1 ;;

[0075] Based on the updated parameters θ 1 , the second support set The second optical image slice and the corresponding second SAR image slice in are input to the first branch and the second branch respectively, to obtain the fifth current eigenvector output by the first branch and the sixth current eigenvector output by the second branch;

[0076] Determine a third loss value based on the fifth current eigenvector, the sixth current eigenvector, and the ternary loss function, and update parameters of the dual-branch network according to the third loss value and the first preset step size;

[0077] According to the updated parameters θ 2 , the second query set The second optical image slice and the corresponding second SAR image slice in are input to the first branch and the second branch respectively, to obtain the seventh current eigenvector output by the first branch and the eighth current eigenvector output by the second branch;

[0078] Determine a fourth loss value based on the fifth current eigenvector, the sixth current eigenvector, and the ternary loss function, and update parameters of the dual-branch network according to the fourth loss value and the second preset step size;

[0079] Determine whether the fourth loss value meets the preset conditions; if so, obtain the trained dual-branch network; if not, set i = i + 1, return to obtain the first amount of training data in any auxiliary scenario, and form the first support set And obtain the second amount of training data in the auxiliary scenario to form the first query set The data set that constitutes the i-th small sample subtask steps.

[0080] Specifically, the first small sample subtask will be constructed The first support set of Input the initialized two-branch network, the first support set Each optical-SAR image slice pair in the first branch and the second branch obtains the corresponding first current feature vector f o and the second current eigenvector f s Based on the first current eigenvector, the second current eigenvector, and the ternary loss function, a first loss value is determined:

[0081]

[0082] Where B represents the number of optical-SAR image slice pairs in each training batch, e a Represents the benchmark sample image slice I a The features obtained by the dual-branch network, e p Represents the matching positive sample image slice I p The corresponding feature, e n It represents the non-matching negative sample image slice I n Corresponding features, margin is a constant greater than 0, representing the threshold parameter that controls the distance between positive sample pairs and negative sample pairs, and d(·) represents the calculation of the Euclidean distance between the two sample features.

[0083] The updated parameter θ' is expressed as follows:

[0084]

[0085] Among them, θ 0 Indicates the initialization parameters, F θ (·) represents the mapping function of the current two-branch network, and α represents the update step size of the gradient descent.

[0086] Next, using the updated parameter θ' and the first query set Continue to train the two-branch network and set the first query set The second optical image slice and the second SAR image slice in are input into the first branch and the second branch respectively to obtain the third current eigenvector and the fourth current eigenvector. The loss value is obtained by the ternary loss function and the parameters of the two-branch network are updated:

[0087]

[0088] Among them, F θ' (·) is the first support set The learned matching model mapping function, β represents the update step size of the gradient descent.

[0089] Furthermore, based on the small sample subtask Based on the dual-branch network obtained through training, cross-scene subtasks are used in turn The second support set of and the second queryset Continue to update the parameters of the dual-branch network. The specific process will not be repeated here.

[0090] Repeat the above steps until the two-branch network converges, and the parameters of the matching model are θ M Finally, after training multiple sets of small-sample subtasks and cross-scenario subtasks, the matching model obtained has sufficient small-sample generalization knowledge and cross-scenario generalization knowledge.

[0091] Optionally, based on the data set of the i-th small sample subtask and the data set of the i-th cross-scenario subtask, alternately training the two-branch network to be trained until the accuracy of the two-branch network reaches a preset condition, and after the step of determining the trained two-branch network as the matching model, the method further includes:

[0092] Optimize and adjust the parameters of the matching model.

[0093] The parameters of the matching model trained by the above steps are now θ M , using the small sample data obtained in a specific scenario to adjust the model parameters θ M Optimize and adjust the model to make it more suitable for the matching task of the target scene. The specific optimization process is as follows:

[0094]

[0095] Where θ M represents the parameters of the matching model, D test a first amount of training data representing a specific scenario, represents the mapping function of the matching model, Represents the ternary loss function, and α represents the first preset step size.

[0096] In step S4, according to the first eigenvector e o With the second eigenvector e s The steps of obtaining the matching result of the optical image and the SAR image by comparing the similarity between the optical image and the SAR image include:

[0097] The first eigenvector e for each first optical image slice o , calculate the Euclidean distance between it and each first SAR image slice respectively;

[0098] The minimum value of the Euclidean distance is determined, and matching is performed with the first SAR image slice corresponding to the minimum value.

[0099] The small sample image matching method based on multi-generalized knowledge accumulation provided by the present invention is further illustrated below through simulation experiments.

[0100] The dataset used in the experiment is the SEN1-2 dataset, which contains spatially aligned optical images and SAR images in a variety of scenarios. The image size in this dataset is 256×256. The SAR images are obtained by observing the Sentinel-1 satellite, and the optical images are obtained by synthesizing the 2nd, 3rd, and 4th band information of the Sentinel-2 satellite. This dataset contains optical-SAR image pairs of multiple scenes in four seasons: spring, summer, autumn, and winter, such as ports, fields, forests, and urban areas.

[0101] The dataset is processed as follows:

[0102] (1) 760 pairs of scene images were randomly selected from different scenes in the dataset, and the SIFT algorithm was used to extract the feature points of each optical image. Image slices of size 64×64 were obtained with each feature point as the center, and image slices of the same size at corresponding positions in the paired SAR images were also obtained.

[0103] (2) According to the subtask construction strategy described above, multiple groups of appropriate first support set and first query set samples are sampled from the optical-SAR image slice pairs of different scenes as the source task data. In a certain scene, some image block pairs are selected as the training set data and test set data of the target task, where the size of the first support set of the source task is set to be consistent with the size of the training set of the target task, and the size of the first query set in the source task is set to be consistent with the size of the test set of the target task.

[0104] During meta-training, the number of training iterations was 10,000, the number of tasks per batch was 32, and the margin of the ternary loss function was set to 0.5. The network was trained using the Adam optimization algorithm, with a learning rate of 0.01 for the inner optimization based on the support data and 0.001 for the outer optimization based on the query data.

[0105] This paper aims to improve matching performance in the case of small sample sizes of target scene data by designing an appropriate small-sample image matching method and accumulating small-sample generalization knowledge and cross-scenario generalization knowledge using data from other scenes. The experimental matching results were evaluated using the False Positive Rate (FPR) 95 (FPR). The lower the FPR95 value, the more accurate the image matching result. The experimental results using different generalization knowledge accumulation methods are shown in Table 1:

[0106] Table 1 Matching performance under different generalized knowledge accumulation methods

[0107]

[0108] As can be seen from Table 1, with the accumulation of the two types of generalized knowledge, the performance of the trained matching model gradually improves. Among them, the matching performance is best when both the small sample generalized knowledge and the cross-scene generalized knowledge accumulation are considered at the same time, indicating that the generalized knowledge accumulation strategy designed in the image matching method provided by the present invention can effectively improve the matching performance.

[0109] It can be seen from the above embodiments that the beneficial effects of the present invention are:

[0110] 1. The matching model constructed in this invention can learn and accumulate effective small-sample generalization knowledge and cross-scenario generalization knowledge from other scenarios, which can be applied to small-sample image matching tasks in specific scenarios, improving matching performance. Compared to the existing technology that directly pre-trains the network using data from other scenarios and then fine-tunes it based on small-sample data from specific scenarios, the generalization knowledge learning strategy designed in this invention ultimately makes the matching model more suitable for small-sample situations in specific scenarios.

[0111] 2. The present invention helps the matching model learn cross-scenario generalization knowledge by constructing cross-scenario subtasks, which can reduce the impact of differences between different scene data on the matching results, and ensure that the final matching model can be well applied to small sample situations in different scenes, thereby improving the versatility of the model to a certain extent.

[0112] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0113] Descriptions with reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0114] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims.

[0115] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the scope of protection of the present invention.

Claims

1. A small sample image matching method based on multi-generalization knowledge accumulation, characterized in that: include: Acquiring a first optical image and a first SAR image to be matched; Preprocessing the first optical image and the first SAR image to obtain a plurality of first optical image slices and a plurality of first SAR image slices; The first optical image slices are input into the first branch of the matching model, and the first SAR image slices are input into the second branch of the matching model to obtain the first eigenvectors and the second eigenvector ; Wherein, the matching model is: a two-branch network trained based on the Few Shot small sample subtask, the cross-scene subtask, the first support set of the small sample subtask, the first query set, and the second support set and second query set of the cross-scene subtask; According to the first eigenvector With the second eigenvector similarity between the optical image and the SAR image, and obtaining a matching result between the optical image and the SAR image; The matching model is trained according to the following steps: Acquire a plurality of optical-SAR image pairs under different scenes from a data set, each of the optical-SAR image pairs comprising a second optical image and a corresponding second SAR image; Preprocess each optical-SAR image pair to form a training data set under the different scenarios ,in, Indicates the n The training data of each scene, n =1,2,……, N , the training data of each scene includes a plurality of second optical image slices of the scene and a plurality of corresponding second SAR image slices; Constructing a two-branch network to be trained and initializing parameters; the two-branch network to be trained includes a first branch and a second branch, wherein the first branch and the second branch each include seven convolutional layers; Any one of the different scenarios is used as a specific scenario and the remaining scenarios are used as auxiliary scenarios. Based on the training data of the specific scenario and the training data of the auxiliary scenario, the first i The dataset of the small sample subtask and i A data set of cross-scenario subtasks; Based on the i The data set of the small sample subtask is the same as the i A data set of cross-scenario subtasks is used to alternately train the two-branch network to be trained until the accuracy of the two-branch network reaches a preset condition, and the trained two-branch network is determined as the matching model; Any one of the different scenarios is used as a specific scenario and the remaining scenarios are used as auxiliary scenarios. Based on the training data of the specific scenario and the training data of the auxiliary scenario, the first i The dataset of the small sample subtask and i The steps for collecting data for a cross-scenario subtask include: Taking any one of the different scenarios as a specific scenario and the remaining scenarios as auxiliary scenarios; Obtain the first amount of training data in any auxiliary scenario to form the first support set , and obtain the second amount of training data in the auxiliary scenario to form the first query set , constituting the i A dataset of small sample subtasks ; wherein the first data amount is smaller than the second data amount; Obtain the first amount of training data in any auxiliary scenario to form the second support set , and obtain the second amount of training data in another auxiliary scenario to form a second query set , constituting the i A dataset of cross-scenario subtasks .

2. The small sample image matching method based on multi-generalization knowledge accumulation according to claim 1 is characterized in that: Preprocess each optical-SAR image pair to form a training data set under the different scenarios The steps include: For each optical-SAR image pair, after extracting feature points in the second optical image using the SIFT algorithm, multiple second optical image slices are obtained by cropping with each feature point as the center, and multiple second SAR image slices are obtained by cropping at the same position of the second SAR image corresponding to the second optical image to form the training data set under the different scenarios. ,in, Indicates the n training data for each scene.

3. The small sample image matching method based on multi-generalization knowledge accumulation according to claim 1, characterized in that: Based on the i The data set of the small sample subtask is the same as the i The steps of alternately training the two-branch network to be trained using a data set of a cross-scenario subtask until the accuracy of the two-branch network reaches a preset condition, and determining the trained two-branch network as a matching model include: The first support set The second optical image slice and the corresponding second SAR image slice in the image are input to the first branch and the second branch respectively, to obtain a first current feature vector output by the first branch and a second current feature vector output by the second branch; Determine a first loss value based on the first current feature vector, the second current feature vector, and a ternary loss function, and update parameters of the dual-branch network according to the first loss value and a first preset step size; Based on the updated parameters , the first query set The second optical image slice and the corresponding second SAR image slice in the image are input to the first branch and the second branch respectively, to obtain a third current eigenvector output by the first branch and a fourth current eigenvector output by the second branch; Based on the third current eigenvector, the fourth current eigenvector and the ternary loss function, a second loss value is determined, and the parameters of the branch network are changed from Updated to ; Based on the updated parameters , the second support set The second optical image slice and the corresponding second SAR image in the image are input to the first branch and the second branch respectively, to obtain the fifth current eigenvector output by the first branch and the sixth current eigenvector output by the second branch; Determining a third loss value based on the fifth current eigenvector, the sixth current eigenvector, and the ternary loss function, and updating parameters of the dual-branch network according to the third loss value and a first preset step size; According to the updated parameters , the second query set The second optical image slice and the corresponding second SAR image in the image are input to the first branch and the second branch respectively, to obtain the seventh current eigenvector output by the first branch and the eighth current eigenvector output by the second branch; Determining a fourth loss value based on the fifth current eigenvector, the sixth current eigenvector, and the ternary loss function, and updating parameters of the dual-branch network according to the fourth loss value and a second preset step size; Determine whether the fourth loss value meets the preset conditions; if so, obtain the trained dual-branch network; if not, set i=i +1, return to obtain the first amount of training data in any auxiliary scenario to form the first support set , and obtain the second amount of training data in the auxiliary scenario to form the first query set , constituting the i A dataset of small sample subtasks steps.

4. The small sample image matching method based on multi-generalized knowledge accumulation according to claim 3 is characterized in that: Based on the i The data set of the small sample subtask is the same as the i The method further comprises the following steps: alternately training the two-branch network to be trained until the accuracy of the two-branch network reaches a preset condition, and determining the trained two-branch network as the matching model. Optimize and adjust the parameters of the matching model.

5. The small sample image matching method based on multi-generalization knowledge accumulation according to claim 4 is characterized in that: The parameters of the matching model are optimized and adjusted according to the following formula: ； Where, represents the parameters of the matching model, training data representing a first amount of data for the specific scenario, represents the mapping function of the matching model, represents the ternary loss function, Indicates the first preset step size.

6. The small sample image matching method based on multi-generalization knowledge accumulation according to claim 1, characterized in that: According to the first eigenvector With the second eigenvector The step of obtaining a matching result between the optical image and the SAR image by comparing the similarity between the optical image and the SAR image comprises: The first eigenvector for each first optical image slice , calculate the Euclidean distance between it and each first SAR image slice respectively; The minimum value of the Euclidean distance is determined, and matching is performed with the first SAR image slice corresponding to the minimum value.

Images