Local descriptor generation method, device, electronic device and computer program product

Abstract

The present invention provides a local descriptor generation method, device, electronic device and computer program product, the method comprising: generating a cross-domain data set, the cross-domain data set comprising multiple pairs of image pairs; obtaining feature-level domain adaptive supervision information according to the scale feature map of the cross-domain data set; obtaining pixel-level cross-domain consistency supervision information and descriptor loss information according to the dense descriptor of the feature-level domain adaptive supervision information; determining the total loss based on the combination of the feature-level domain adaptive supervision information, the pixel-level cross-domain consistency supervision information and the descriptor loss information, and using the total loss for supervised training of the network to obtain the local descriptor. The present invention can enhance the invariance and robustness of the local descriptor and improve the accuracy of the descriptor in the image matching task.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a local descriptor generation method, device, electronic device and computer program product. Background Art

[0002] At present, extracting accurate and efficient local feature descriptors is an essential processing step for various computer vision applications, such as mobile visual search, visual tracking, image matching, image retrieval, and visual localization. Among the traditional manual feature extraction methods, SIFT plays a vital role in computer vision tasks with scale invariance and rotation invariance. The progress of deep learning has made the learned local descriptors far superior to the classical methods. In recent years, models that jointly learn keypoint detection and descriptors have achieved better performance in image matching and visual localization. Unlike patch-based feature descriptors, these methods use fully convolutional neural networks to extract dense feature descriptors for the entire image instead. For deep learning-based algorithms, extracting dense feature descriptors for the entire image has become a trend in recent years. In general, robust local descriptors require invariance, that is, when the illumination or viewpoint of the same keypoint changes, their descriptors can still be successfully matched.

[0003] In the prior art, in order to accurately match in various application scenarios, the descriptor needs to capture as much invariance as possible. However, extracting robust descriptors for complex scenes including day-night changes and seasonal changes is extremely challenging and can easily lead to mismatches. Existing methods enhance the robustness of descriptors by utilizing large training datasets and complex network structures, but this inevitably brings about a large amount of computation and limits the application of descriptors. Summary of the invention

[0004] The present invention provides a local descriptor generation method, device, electronic device and computer program product, which are used to solve the defects of low invariance and robustness of descriptors in the prior art and improve the accuracy of descriptors in image matching tasks.

[0005] The present invention provides a local descriptor generation method, comprising:

[0006] Generate a cross-domain dataset, wherein the cross-domain dataset includes multiple pairs of images;

[0007] Acquiring feature-level domain adaptive supervision information according to the scale feature map of the cross-domain dataset;

[0008] Obtain pixel-level cross-domain consistency supervision information and descriptor loss information according to the dense descriptor of the feature-level domain adaptive supervision information;

[0009] Based on the combination of the feature-level domain adaptation supervision information, the pixel-level cross-domain consistency supervision information and the descriptor loss information, a total loss is determined, and the total loss is used for supervised training of the network to obtain a local descriptor.

[0010] According to the local descriptor generation method provided by the present invention, the generation of a cross-domain dataset includes:

[0011] Determine the source domain image I based on the dataset S ;

[0012] According to the source domain image I S , get the corresponding target domain image I T , the target domain image I T and source domain image I S Constructing image pairs;

[0013] According to the target domain image I T With the source domain image I S , obtaining the label of the supervised training of the network, wherein the cross-domain dataset includes the image pair and the label of the supervised training of the network, and the target domain image I T With the source domain image I S The corresponding relationships include style transfer relationship, Homomorphic transformation relationship and real annotation relationship.

[0014] According to the local descriptor generation method provided by the present invention, the step of obtaining feature-level domain adaptive supervision information according to the scale feature map of the cross-domain dataset includes:

[0015] Inputting the image pair into an encoder to obtain a plurality of scale feature maps;

[0016] The scale feature map is subjected to domain adversarial learning through a domain classifier to extract global domain invariant features, wherein the domain classifier includes a gradient reversal layer and at least one fully connected layer;

[0017] A feature-level domain loss function is calculated according to the scale feature map to enhance the performance of the global domain invariant feature, wherein the feature-level domain loss function is used to characterize the feature-level domain adaptive supervision information.

[0018] According to the local descriptor generation method provided by the present invention, the scale feature map is subjected to domain adversarial learning by a domain classifier to extract global domain invariant features, including:

[0019] Calculating a source domain classification loss value according to the hidden layer and the classification layer of the domain classifier;

[0020] Calculate the domain classification loss value;

[0021] A gradient maximization loss function is calculated according to the source domain classification loss value and the domain classification loss value, so as to improve the performance of the domain classifier by minimizing the source domain classification loss value and maximizing the domain classification loss value.

[0022] According to the local descriptor generation method provided by the present invention, the dense descriptor based on the feature-level domain adaptive supervision information obtains pixel-level cross-domain consistency supervision information and descriptor loss information, including:

[0023] Acquire a feature map through a decoder according to the global domain invariant feature and the multiple scale feature maps;

[0024] According to the feature map, a dense descriptor is obtained;

[0025] Determine a mean feature map, a maximum feature map, a positive distance, and a negative distance according to the dense descriptor;

[0026] Calculating a pixel-level cross-domain consistency loss function according to the mean feature map and the maximum feature map of the dense descriptor, wherein the pixel-level cross-domain consistency loss function is used to characterize the pixel-level cross-domain consistency supervision information;

[0027] A triplet loss function is calculated according to the positive distance and the negative distance of the dense descriptor, wherein the triplet loss function is used to characterize the descriptor loss information.

[0028] According to the local descriptor generation method provided by the present invention, the total loss is determined based on the combination of the feature-level domain adaptive supervision information, the pixel-level cross-domain consistency supervision information and the descriptor loss information, including:

[0029] Determining a total loss according to a weighted sum of the feature-level domain adaptation supervision information, the pixel-level cross-domain consistency supervision information, and the descriptor loss information;

[0030] Network training is performed according to the total loss to obtain a local descriptor for enhancing the invariance of the local descriptor, wherein the network includes an encoder-decoder network.

[0031] The present invention also provides a local descriptor generating device, comprising:

[0032] Generation module, used to generate cross-domain datasets;

[0033] A first acquisition module is used to acquire feature-level domain adaptive supervision information according to the scale feature map of the cross-domain dataset;

[0034] A second acquisition module is used to obtain pixel-level cross-domain consistency supervision information and descriptor loss information according to the dense descriptor of the feature-level domain adaptive supervision information;

[0035] A determination module is used to determine the total loss based on the combination of the feature-level domain adaptive supervision information, the pixel-level cross-domain consistency supervision information and the descriptor loss information, and the total loss is used for supervised training of the network to obtain local descriptors.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the local descriptor generation methods described above is implemented.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the local descriptor generation method as described above is implemented.

[0038] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the local descriptor generation method as described above is implemented.

[0039] The local descriptor generation method, device, electronic device and computer program product provided by the present invention quickly construct a diversified domain data set for network training through a cross-domain data set; reduce the difference between the feature domains of the image pair by calculating the feature-level domain adaptation loss, thereby improving the robustness of the descriptor; compensate for the inconsistency between the descriptors corresponding to the pixel key points by calculating the pixel-level cross-domain consistency loss, thereby enhancing the invariance of the descriptor; determine the total loss by combining the feature-level domain adaptation loss, the pixel-level cross-domain consistency loss and the descriptor loss, and train the network through the total loss to obtain the local descriptor, thereby enhancing the invariance of the local descriptor while ensuring the discrimination ability, thereby improving the accuracy of the local descriptor in the image matching task. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is one of the flow charts of the local descriptor generation method provided by the present invention;

[0042] Figure 2 It is a schematic diagram of the network structure of the scale feature map of the local descriptor generation method provided by the present invention;

[0043] Figure 3This is the second flow chart of the local descriptor generation method provided by the present invention;

[0044] Figure 4 It is a schematic diagram of feature-level domain adaptive loss calculation of the local descriptor generation method provided by the present invention;

[0045] Figure 5 This is the third flow chart of the local descriptor generation method provided by the present invention;

[0046] Figure 6 It is a schematic diagram of calculating pixel-level cross-domain consistency loss of the local descriptor generation method provided by the present invention;

[0047] Figure 7 This is the fourth flow chart of the local descriptor generation method provided by the present invention;

[0048] Figure 8 It is a schematic diagram of total loss calculation of the local descriptor generation method provided by the present invention;

[0049] Fig. 9 It is a structural schematic diagram of a local descriptor generating device provided by the present invention;

[0050] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Combine the following Figure 1-Figure 8 The local descriptor generation method of the present invention is described.

[0053] Figure 1 It is one of the flowcharts of the local descriptor generation method provided by the present invention, such as Figure 1 As shown, the method includes:

[0054] Step 110: Generate a cross-domain dataset, where the cross-domain dataset includes multiple image pairs.

[0055] Optionally, generate a cross-domain dataset, including:

[0056] Determine the source domain image I based on the dataset S .

[0057] Optionally, the Aachen-Day-Night dataset and MegaDepth dataset commonly used in image matching and visual positioning tasks are selected, and all 4479 images of the Aachen-Day-Night dataset and 11800 image pairs selected in MegaDepth are used as source domain images I S , to build a comprehensive cross-domain dataset for training.

[0058] According to the source domain image I S , get the corresponding target domain image I T , target domain image I T and source domain image I S Constitute an image pair, that is, each image pair includes a source domain image I S and a target domain image I T .

[0059] According to the target domain image I T With the source domain image I S The corresponding relationship between the target domain image I and the supervised training label of the network is obtained. The cross-domain dataset includes the image pair and the supervised training label of the network. T With the source domain image I S The corresponding relationships include style transfer relationship, Homomorphic transformation relationship and real annotation relationship.

[0060] Optionally, the source domain image I is transferred using the style transfer method HIDM S The image domains are converted to morning, noon, dusk and night respectively. At the same time, all the style-converted images are subjected to Homomorphic transformation with a certain probability to increase the viewpoint change. S corresponds to the result of the Homomorphic transformation process or corresponds to an image with style transfer only. S The corresponding image is regarded as the target domain image I T , the target domain image I T With the source domain image I S The correspondence between them is used as the label required for descriptor supervision, and 32558 image pairs are randomly selected as the cross-domain training dataset.

[0061] Optionally, if the source domain image I S From the MegaDepth dataset, source domain image I S It also corresponds to images with true labeled correspondences, whose correspondences have been given in the dataset.

[0062] Optionally, if the source domain image I SAfter the Homomorphic transformation, the transformed image is divided into 20×20 grids, and a point is randomly and uniformly sampled in each grid. The corresponding points are obtained according to the homography transformation, that is, according to the Homomorphic transformation, and the correspondence between these image pairs is used as the annotation required for descriptor supervision.

[0063] like Figure 1 As shown, in step 120, feature-level domain adaptive supervision information is obtained according to the scale feature map of the cross-domain dataset.

[0064] Figure 2 is a schematic diagram of the network structure of the scale feature map of the local descriptor generation method provided by the present invention, Figure 3 This is the second flow chart of the local descriptor generation method provided by the present invention. Figure 4 It is a schematic diagram of feature-level domain adaptive loss calculation of the local descriptor generation method provided by the present invention.

[0065] Alternatively, if Figure 2 As shown, according to the scale feature map of the cross-domain dataset, feature-level domain adaptive supervision information is obtained, including:

[0066] like Figure 2 As shown, in step 210, the image pair is input into an encoder to obtain multiple scale feature maps.

[0067] Optionally, for each input image, Resnet is used as an encoder to extract global domain invariant features. The Resnet encoder generates multiple feature maps of different sizes, with sizes of 64×64, 128×128, 256×256, and 512×512.

[0068] like Figure 2 As shown, in step 220, the scale feature map is subjected to domain adversarial learning through a domain classifier to extract global domain invariant features, wherein the domain classifier includes a gradient reversal layer and at least one fully connected layer.

[0069] like Figure 3 As shown, optionally, for the scale feature map with a scale of 64×64, a gradient reversal layer is applied to realize domain adversarial learning, that is, in the forward propagation, the features of the gradient reversal layer remain unchanged, and in the back propagation, the gradient returned by the gradient reversal layer maximizes the loss function, and the parameters are updated to the negative gradient direction, so as to reduce the domain adaptation neural network DANN to the target domain image I T and source domain image I S The discriminative ability of the two images of the image pair is improved, so that the two images of the image pair are mapped to the same feature domain as much as possible, and at the same time, the performance of the domain adaptation neural network is improved.

[0070] like Figure 3As shown, optionally, the scale feature map is subjected to domain adversarial learning through a domain classifier, specifically including:

[0071] The domain adaptation neural network DANN is used, and three fully connected layers and one gradient flipping layer are added to the DANN feedforward model to obtain global domain invariant features for prediction. Among them, the objective function of the domain adaptation neural network DANN includes: source domain classification loss and domain classification loss.

[0072] Among them, given an example (x i ,y i ), through the hidden layer G h and the classification layer G y , calculate the source domain classification loss, the source domain classification loss is shown in formula (1):

[0073]

[0074] The domain classification loss makes it difficult for the domain discriminator to distinguish the target domain image I T With the source domain image I S , so that the source domain image I S and the target domain image I T Aligned in distribution, the domain classification loss is shown in formula (2):

[0075]

[0076] Among them, b i is a label, if x i When b is the source domain i is 0, if x i When b is the target domain i is 1, i is used to represent the i-th pixel.

[0077] The domain adaptation neural network DANN is specifically used to minimize the source domain classification loss and maximize the domain classification loss. Therefore, the gradient maximization loss function is shown in formula (3):

[0078]

[0079] like Figure 4 As shown, step 230 calculates a feature-level domain loss function based on the scale feature map to enhance the performance of the global domain invariant features, wherein the feature-level domain loss function is used to characterize the feature-level domain adaptive supervision information.

[0080] like Figure 4 As shown, optionally, the feature level domain loss function calculation method includes:

[0081] Calculate the Shannon entropy that minimizes the target sample prediction so that the model produces a high-confidence prediction. The confidence is shown in formula (4):

[0082] H(p i )=(-p i log(p i )-(1-p i )log(1-p i )) (4),

[0083] Among them, p i is the target domain image I T The domain prediction score of .

[0084] The value H after the tanh activation function is calculated according to Shannon entropy. The value H is shown in formula (5):

[0085] H = tanh(H(p i )) (5);

[0086] Calculate the cross entropy CE, which is shown in formula (6):

[0087]

[0088] Among them, l i is the domain category label of the image, l i =1 indicates the target domain image I T The label, l i =0 means source domain image I S Labels;

[0089] Calculate the feature-level domain adaptive loss function, which is shown in formula (7):

[0090] L feat (F(I S ),F(I T ))=(CE+H) 2 (7)

[0091] Among them, F(I S ) and F(I T ) represent the source domain image feature map and target domain image feature map with an input scale of 64×64 respectively.

[0092] like Figure 1 As shown, step 130 obtains pixel-level cross-domain consistency supervision information and descriptor loss information based on the dense descriptor of the feature-level domain adaptive supervision information.

[0093] Figure 5 This is the third flow chart of the local descriptor generation method provided by the present invention. Figure 6 Schematic diagram of pixel-level cross-domain consistency loss calculation of the local descriptor generation method provided by the present invention. Figure 5-Figure 6As shown, based on the dense descriptor of feature-level domain adaptive supervision information, pixel-level cross-domain consistency supervision information and descriptor loss information are obtained, including:

[0094] Step 310: Obtain a feature map through a decoder according to the global domain invariant features and multiple scale feature maps.

[0095] Optionally, an FPN structure is used as a decoder, specifically including:

[0096] The four scale feature maps output by the Resnet encoder are adjusted through 1×1 convolution, namely the channels of the scale feature maps of 64×64, 128×128, 256×256 and 512×512 respectively; the above scale feature maps are added to the nearest neighbor upsampling result of the low-resolution feature map to obtain the final feature map.

[0097] Step 320: Obtain a dense descriptor according to the feature map.

[0098] Optionally, according to the final feature map, a 1×1 convolutional layer is used to obtain a 128-dimensional dense descriptor, specifically including: given an image pair (I S , I T ), the source domain image I S and the target domain image I T Input encoder-decoder structure, and finally get dense descriptor (D S ,D T ).

[0099] Step 330, determining a mean feature map, a maximum feature map, a positive distance, and a negative distance according to the dense descriptor.

[0100] Optionally, calculate the mean of each descriptor so that each image pair can obtain a pair of descriptor mean feature maps The descriptor mean feature map for the i-th pixel is shown in formula (8):

[0101]

[0102] Calculate the maximum value of each descriptor so that each image pair can obtain the maximum feature map of a pair of descriptors ((D S ) max ,(D T ) max ), the maximum feature map of the descriptor of the i-th pixel is shown in formula (9):

[0103] d i max =(d i S ) max -(d iT ) max (9);

[0104] Optionally, the image pair (I S , I T ) The corresponding dense descriptors at the i-th pixel are and Then the positive distance p i As shown in formula (10):

[0105] p i =||d i S -d i T ||2 (10),

[0106] Negative distance n i As shown in formula (11):

[0107]

[0108] Step 340, calculating a pixel-level cross-domain consistency loss function based on the mean feature map and the maximum feature map of the dense descriptor, wherein the pixel-level cross-domain consistency loss function is used to characterize the pixel-level cross-domain consistency supervision information.

[0109] Optionally, the sampled descriptor mean feature map and the maximum feature map are used as the input of the pixel-level cross-domain consistency loss function for supervision. Preferably, considering the source domain image I S and the target domain image I T The appearance or style may be different, but the descriptors corresponding to the key points of the two images are close enough, so the pixel-level cross-domain consistency loss function is shown in formula (12):

[0110]

[0111] Among them, i is the i-th pixel, and d i max is the descriptor mean feature map and maximum feature map for the i-th pixel, and n is the total number of pixels in an image.

[0112] Step 350 , calculating a triplet loss function according to the positive distance and the negative distance of the dense descriptor, wherein the triplet loss function is used to characterize the descriptor loss information.

[0113] Optionally, given a positive distance p i and negative distance n i , calculate the triplet loss function, and use the triplet loss function that is conducive to distinguishing outliers as the descriptor loss function. The triplet loss function is shown in formula (13):

[0114] L des =max(0,p i -n i +1) (13),

[0115] Among them, the goal of the triplet loss function, that is, the descriptor loss function, is to make the descriptors of the same label (that is, positive examples) as close as possible in space, and at the same time, the descriptors of different labels (that is, negative examples) are as far away as possible in space. Therefore, the negative example should be at least 1 farther away from the positive example.

[0116] like Figure 1 As shown, step 140 determines the total loss based on the combination of feature-level domain adaptive supervision information, pixel-level cross-domain consistency supervision information and descriptor loss information. The total loss is used for supervised training of the network to obtain local descriptors.

[0117] Figure 7 FIG. 8 is a schematic diagram of the total loss calculation of the local descriptor generation method provided by the present invention. Figure 7 - Figure 8 As shown in FIG, based on the combination of feature-level domain adaptive supervision information, pixel-level cross-domain consistency supervision information and triplet loss information, descriptor supervision processing is performed, including:

[0118] Step 410, determining the total loss according to the weighted sum of the feature-level domain adaptation supervision information, the pixel-level cross-domain consistency supervision information and the descriptor loss information.

[0119] Combining the above triplet loss function, feature-level domain adaptation loss function and pixel-level cross-domain consistency loss function, the total loss is calculated as shown in formula (14):

[0120] L total =α·L feat +βL consis +λ·L des (14)

[0121] Among them, the weights α, β, and λ are all 1.

[0122] Step 420 , performing network training according to the total loss to obtain local descriptors for enhancing the invariance of the local descriptors, wherein the network includes an encoder-decoder network, namely a Resnet-FPN network.

[0123] Optionally, the network is trained based on the total loss to obtain a local descriptor based on domain adaptation, which is used in many downstream tasks such as image matching, image retrieval, and visual localization. The local descriptor based on domain adaptation is extremely robust and can obtain accurate image matching results. The local descriptor based on domain adaptation has superiority in image matching tasks.

[0124] The local descriptor generation method provided by the present invention uses a cross-domain data set to quickly build a diversified domain data set for network training; by calculating the feature-level domain adaptive loss, the difference between the feature domains of the image pair is reduced to improve the robustness of the descriptor; by calculating the pixel-level cross-domain consistency loss, the inconsistency between the descriptors corresponding to the pixel key points is compensated, thereby enhancing the invariance of the descriptor; combining the feature-level domain adaptive loss, the pixel-level cross-domain consistency loss and the descriptor loss to determine the total loss, and using the total loss to train the network to obtain the local descriptor, thereby enhancing the invariance of the local descriptor while ensuring the distinguishing ability, thereby improving the accuracy of the local descriptor in the image matching task.

[0125] The local descriptor generating device provided by the present invention is described below. The local descriptor generating device described below and the local descriptor generating method described above can be referred to each other.

[0126] The present invention provides a local descriptor generating device. Fig. 9 A schematic diagram of the structure of the local descriptor generation device provided by the present invention, such as Fig. 9 As shown, the local descriptor generating device 500 includes: a construction module 501, a first acquisition module 502, a second acquisition module 503 and a calculation module 504; wherein,

[0127] Construction module 501, used to generate a cross-domain dataset;

[0128] A first acquisition module 502 is used to acquire feature-level domain adaptive supervision information according to a scale feature map of a cross-domain dataset;

[0129] The second acquisition module 503 is used to obtain pixel-level cross-domain consistency supervision information and descriptor loss information according to the dense descriptor of the feature-level domain adaptive supervision information;

[0130] The calculation module 504 is used to determine the total loss based on the combination of feature-level domain adaptive supervision information, pixel-level cross-domain consistency supervision information and descriptor loss information. The total loss is used for supervised training of the network to obtain local descriptors.

[0131] The local descriptor generation device provided by the present invention uses a cross-domain data set to quickly build a diversified domain data set for network training; by calculating the feature-level domain adaptive loss, the difference between the feature domains of the image pair is reduced to improve the robustness of the descriptor; by calculating the pixel-level cross-domain consistency loss, the inconsistency between the descriptors corresponding to the pixel key points is compensated, thereby enhancing the invariance of the descriptor; combining the feature-level domain adaptive loss, the pixel-level cross-domain consistency loss and the descriptor loss to determine the total loss, and using the total loss to train the network to obtain the local descriptor, thereby enhancing the invariance of the local descriptor while ensuring the ability to distinguish, thereby improving the accuracy of the local descriptor in the image matching task.

[0132] Optionally, the construction module 501 is specifically used to:

[0133] Determine the source domain image I based on the dataset S ;

[0134] According to the source domain image I S , get the corresponding target domain image I T , target domain image I T and source domain image I S Constructing image pairs;

[0135] According to the target domain image I T With the source domain image I S The corresponding relationship between the target domain image I and the supervised training label of the network is obtained. The cross-domain dataset includes the image pair and the supervised training label of the network. T With the source domain image I S The corresponding relationships include style transfer relationship, Homomorphic transformation relationship and real annotation relationship.

[0136] Optionally, the first acquisition module 502 is specifically configured to:

[0137] Input the image pair into the encoder to obtain multiple scale feature maps;

[0138] The scale feature map is subjected to domain adversarial learning through a domain classifier to extract global domain invariant features, wherein the domain classifier includes a gradient reversal layer and at least one fully connected layer;

[0139] The feature-level domain loss function is calculated according to the scale feature map to enhance the performance of the global domain invariant features, wherein the feature-level domain loss function is used to characterize the feature-level domain adaptive supervision information.

[0140] Optionally, the first acquisition module 502 is specifically configured to:

[0141] Calculate the source domain classification loss value based on the hidden layer and classification layer of the domain classifier;

[0142] Calculate the domain classification loss value;

[0143] According to the source domain classification loss value and the domain classification loss value, a gradient maximization loss function is calculated to improve the performance of the domain classifier by minimizing the source domain classification loss value and maximizing the domain classification loss value.

[0144] Optionally, the second acquisition module 503 is specifically configured to:

[0145] According to the global domain invariant features and multiple scale feature maps, a feature map is obtained through a decoder;

[0146] According to the feature map, obtain the dense descriptor;

[0147] Determine the mean feature map, the maximum feature map, the positive distance and the negative distance based on the dense descriptor;

[0148] According to the mean feature map and the maximum feature map of the dense descriptor, a pixel-level cross-domain consistency loss function is calculated, wherein the pixel-level cross-domain consistency loss function is used to characterize the pixel-level cross-domain consistency supervision information;

[0149] According to the positive distance and negative distance of the dense descriptor, a triplet loss function is calculated, where the triplet loss function is used to characterize the descriptor loss information.

[0150] Optionally, the calculation module 504 is specifically configured to:

[0151] Determine the total loss based on the weighted sum of feature-level domain adaptation supervision information, pixel-level cross-domain consistency supervision information, and descriptor loss information;

[0152] The network is trained according to the total loss to obtain a local descriptor for enhancing the invariance of the local descriptor. The network includes an encoder-decoder network.

[0153] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device 600 may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the local descriptor generation method, which includes:

[0154] Generate a cross-domain dataset, which includes multiple pairs of images;

[0155] According to the scale feature map of the cross-domain dataset, feature-level domain adaptive supervision information is obtained;

[0156] Based on the dense descriptor of feature-level domain adaptive supervision information, pixel-level cross-domain consistency supervision information and descriptor loss information are obtained;

[0157] Based on the combination of feature-level domain adaptation supervision information, pixel-level cross-domain consistency supervision information and descriptor loss information, the total loss is determined and used for supervised training of the network to obtain local descriptors.

[0158] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0159] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the local descriptor generation method provided by the above methods, the method includes:

[0160] Generate a cross-domain dataset, which includes multiple pairs of images;

[0161] According to the scale feature map of the cross-domain dataset, feature-level domain adaptive supervision information is obtained;

[0162] Based on the dense descriptor of feature-level domain adaptive supervision information, pixel-level cross-domain consistency supervision information and descriptor loss information are obtained;

[0163] Based on the combination of feature-level domain adaptation supervision information, pixel-level cross-domain consistency supervision information and descriptor loss information, the total loss is determined and used for supervised training of the network to obtain local descriptors.

[0164] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a local descriptor provided by the above methods is implemented, and the method comprises:

[0165] Generate a cross-domain dataset, which includes multiple pairs of images;

[0166] According to the scale feature map of the cross-domain dataset, feature-level domain adaptive supervision information is obtained;

[0167] Based on the dense descriptor of feature-level domain adaptive supervision information, pixel-level cross-domain consistency supervision information and descriptor loss information are obtained;

[0168] Based on the combination of feature-level domain adaptation supervision information, pixel-level cross-domain consistency supervision information and descriptor loss information, the total loss is determined and used for supervised training of the network to obtain local descriptors.

[0169] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A local descriptor generation method, characterized in that: include: Generate a cross-domain dataset, wherein the cross-domain dataset includes multiple pairs of images; Acquiring feature-level domain adaptive supervision information according to the scale feature map of the cross-domain dataset; Obtain pixel-level cross-domain consistency supervision information and descriptor loss information according to the dense descriptor of the feature-level domain adaptive supervision information; Determine a total loss based on a combination of the feature-level domain adaptation supervision information, the pixel-level cross-domain consistency supervision information and the descriptor loss information, wherein the total loss is used for supervised training of the network to obtain a local descriptor; The step of obtaining feature-level domain adaptive supervision information according to the scale feature map of the cross-domain dataset includes: Inputting the image pair into an encoder to obtain a plurality of scale feature maps; The scale feature map is subjected to domain adversarial learning through a domain classifier to extract global domain invariant features, wherein the domain classifier includes a gradient reversal layer and at least one fully connected layer; Calculating a feature-level domain loss function according to the scale feature map to enhance the performance of the global domain invariant feature, wherein the feature-level domain loss function is used to characterize the feature-level domain adaptive supervision information; The obtaining pixel-level cross-domain consistency supervision information and descriptor loss information according to the dense descriptor of the feature-level domain adaptive supervision information includes: Acquire a feature map through a decoder according to the global domain invariant feature and the multiple scale feature maps; According to the feature map, a dense descriptor is obtained; Determine a mean feature map, a maximum feature map, a positive distance, and a negative distance according to the dense descriptor; Calculating a pixel-level cross-domain consistency loss function according to the mean feature map and the maximum feature map of the dense descriptor, wherein the pixel-level cross-domain consistency loss function is used to characterize the pixel-level cross-domain consistency supervision information; A triplet loss function is calculated according to the positive distance and the negative distance of the dense descriptor, wherein the triplet loss function is used to characterize the descriptor loss information.

2. The local descriptor generation method according to claim 1, characterized in that: The generating of the cross-domain dataset comprises: Determine the source domain image I based on the dataset S ; According to the source domain image I S , get the corresponding target domain image I T , the target domain image I T and source domain image I S Constructing image pairs; According to the target domain image I T With the source domain image I S , obtaining the label of the supervised training of the network, wherein the cross-domain dataset includes the image pair and the label of the supervised training of the network, and the target domain image I T With the source domain image I S The corresponding relationships include style transfer relationship, Homomorphic transformation relationship and real annotation relationship.

3. The local descriptor generation method according to claim 1, characterized in that: The scale feature map is subjected to domain adversarial learning through a domain classifier to extract global domain invariant features, including: Calculating a source domain classification loss value according to the hidden layer and the classification layer of the domain classifier; Calculate the domain classification loss value; A gradient maximization loss function is calculated according to the source domain classification loss value and the domain classification loss value, so as to improve the performance of the domain classifier by minimizing the source domain classification loss value and maximizing the domain classification loss value.

4. The local descriptor generation method according to claim 1, characterized in that: Determining a total loss based on a combination of the feature-level domain adaptation supervision information, the pixel-level cross-domain consistency supervision information, and the descriptor loss information includes: Determining a total loss according to a weighted sum of the feature-level domain adaptation supervision information, the pixel-level cross-domain consistency supervision information, and the descriptor loss information; Network training is performed according to the total loss to obtain a local descriptor for enhancing the invariance of the local descriptor, wherein the network includes an encoder-decoder network.

5. A local descriptor generation device, characterized in that: include: A generating module, configured to generate a cross-domain dataset, wherein the cross-domain dataset includes a plurality of image pairs; A first acquisition module is used to acquire feature-level domain adaptive supervision information according to the scale feature map of the cross-domain dataset; A second acquisition module is used to obtain pixel-level cross-domain consistency supervision information and descriptor loss information according to the dense descriptor of the feature-level domain adaptive supervision information; A determination module, configured to determine a total loss based on a combination of the feature-level domain adaptation supervision information, the pixel-level cross-domain consistency supervision information, and the descriptor loss information, wherein the total loss is used for supervised training of the network to obtain a local descriptor; The first acquisition module is specifically used to: Inputting the image pair into an encoder to obtain a plurality of scale feature maps; The scale feature map is subjected to domain adversarial learning through a domain classifier to extract global domain invariant features, wherein the domain classifier includes a gradient reversal layer and at least one fully connected layer; Calculating a feature-level domain loss function according to the scale feature map to enhance the performance of the global domain invariant feature, wherein the feature-level domain loss function is used to characterize the feature-level domain adaptive supervision information; The second acquisition module is specifically used for: Acquire a feature map through a decoder according to the global domain invariant feature and the multiple scale feature maps; According to the feature map, a dense descriptor is obtained; Determine a mean feature map, a maximum feature map, a positive distance, and a negative distance according to the dense descriptor; Calculating a pixel-level cross-domain consistency loss function according to the mean feature map and the maximum feature map of the dense descriptor, wherein the pixel-level cross-domain consistency loss function is used to characterize the pixel-level cross-domain consistency supervision information; A triplet loss function is calculated according to the positive distance and the negative distance of the dense descriptor, wherein the triplet loss function is used to characterize the descriptor loss information.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the local descriptor generation method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the local descriptor generation method according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the local descriptor generation method according to any one of claims 1 to 4 is implemented.

Images