Local feature detection description method based on multi-scale semantic feature fusion

Abstract

The invention discloses a local feature detection description method based on multi-scale semantic feature fusion, and the method comprises the steps: firstly, generating a preliminary feature descriptor through homography transformation and pyramid feature extraction normal forms in combination with dynamic weight fusion; performing multi-dimensional feature fusion on the image by using parallel convolution and self-attention, and extracting richer local descriptor information from coarse to fine and from local to global; and then the robustness and discrimination capability of the features are optimized by introducing a semantically weighted triple loss function, and the global representation capability of the features is further improved. And finally, category-known global mask information is generated by using a semantic segmentation network to guide feature point screening, so that global features are more effectively captured, and interference of dynamic objects is reduced. According to the method, the feature detection description effect in a complex scene is remarkably improved, high calculation efficiency and accuracy are achieved, and the method has important practical significance on technical application of robot navigation, unmanned driving and the like.

Description

Technical Field

[0001] The present invention relates to a local feature detection and description method based on multi-scale semantic feature fusion, and belongs to the technical field of computer image processing. Background Art

[0002] Local feature detection and description are crucial tasks in computer vision and are widely used in multiple fields, including simultaneous localization and mapping (SLAM), visual positioning, image retrieval and matching, and 3D reconstruction. Local features refer to key points or regions in an image and are usually extracted by feature detection algorithms. These features can effectively describe unique local information in an image. Common local feature detection algorithms include SIFT, SURF, and ORB. Hand-crafted local feature methods usually rely on gradient changes in image intensity, can identify key points with strong stability, and describe and model local information. However, these hand-crafted feature-based methods face many challenges when dealing with complex scenes.

[0003] As image appearance changes and scene complexity increases, local feature detection and matching tasks face many new challenges. For example, weakly textured areas (such as large flat surfaces), repetitive textures (such as building grilles), lighting changes (such as the transition between day and night), seasonal changes (such as the visual difference between winter and summer), and dynamic objects in the scene (such as pedestrians and vehicles) all pose challenges to feature extraction. These issues make traditional hand-crafted feature methods ineffective in these complex environments.

[0004] In recent years, data-driven convolutional neural network (CNN) architectures have demonstrated significant advantages in local feature detection and description tasks, particularly in coping with image appearance variations such as lighting, perspective, and scene changes. Using large amounts of training data, CNNs can automatically learn robust features and effectively identify stable elements in images. Methods such as SuperPoint, R2D2, DISK, and D2Net have made local feature extraction more efficient and accurate.

[0005] However, convolutional networks also face some limitations, especially in special scenarios. Convolutional networks gradually stack local receptive fields to construct higher-level feature representations. This method is suitable for extracting local features. However, its limitation is that for areas with a simple geometric structure or lack of obvious texture features (such as water bodies, sky, etc.), convolutional networks find it difficult to capture globally reliable feature information. These areas lack significant local features, resulting in poor feature matching and positioning performance of the network in these scenes. In addition, convolutional networks also face the problem of interference from dynamic objects in visual positioning tasks, such as pedestrians, vehicles or other dynamic objects (such as fish). These dynamic objects will introduce unstable feature information and interfere with the network's feature extraction process. Since convolutional neural networks mainly rely on local features in static scenes, they fail to effectively distinguish the interference brought by these dynamic objects, resulting in poor positioning performance in dynamic environments, especially in tasks with high real-time requirements, such as robot visual navigation and unmanned driving. Therefore, there is an urgent need for a local feature detection and description method that can enhance the capture of reliable feature information and high robustness. Summary of the Invention

[0006] The content of this application is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this application is not intended to identify key features or essential features of the technical solution for which protection is sought, nor is it intended to limit the scope of the technical solution for which protection is sought.

[0007] In response to the problems and shortcomings of the existing technology, the present invention aims to provide a local feature detection and description method based on multi-scale semantic feature fusion, which includes four stages: dataset preprocessing, multi-scale feature extraction, descriptor feature refinement, and semantic confidence guidance. Through multi-scale coarse-to-fine descriptor optimization and prior semantic information guidance, the network can simultaneously focus on local and global information, improving the precision of downstream visual tasks, thereby achieving better robustness and accuracy. This is used to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The present invention discloses a local feature detection and description method based on multi-scale semantic feature fusion, comprising the following steps:

[0010] Step 1: Obtain an image to be processed and perform homography transformation on the image;

[0011] Step 2: Input the homography-transformed image into the multi-scale backbone network to extract the detection head, descriptor head, weight head, and confidence head;

[0012] Step 3, using convolution and self-attention to concurrently refine the descriptor head to generate a descriptor;

[0013] Step 4: Combine prior semantic information to set the image category confidence and guide the neglect of features on unstable objects;

[0014] Step 5: Introduce the descriptor total loss function to guide the confidence header generation.

[0015] Preferably, the steps of step 2 include the following steps:

[0016] Step 2.1, the image after homography transformation is mapped through a multi-scale backbone network to obtain four intermediate layer features;

[0017] Step 2.2: Fuse the last three intermediate layer features with the first intermediate layer feature in the channel dimension.

[0018] Step 2.3: Output the detection head, descriptor head, weight head, and confidence head through the fused feature map.

[0019] Preferably, the dynamic weighted feature fusion in step 2.2 further includes the following steps:

[0020] Step 2.2.1, stacking the four intermediate layer features on the channel to obtain a feature map;

[0021] Step 2.2.2, linearly transform the feature map through the attention mechanism to generate the corresponding query Q, key K and value V;

[0022] Step 2.2.3, dynamically calculate the attention weight using the query Q, key K and value V;

[0023] Step 2.2.4, performing weighted summation on the feature map using the attention weights;

[0024] In step 2.2.5, the weighted summed feature map is used to generate a heat map for observation.

[0025] Preferably, in step 3, vectors Q, K, and V are first generated by operating with three 1×1 convolution kernels, and the following split-fusion steps are included for the convolution branches:

[0026] Step 3.1.1, concatenate the vectors Q, K, and V to obtain a fusion vector;

[0027] Step 3.1.2, generating corresponding weights by passing the fusion vector through the channel attention branch side and the spatial attention branch side;

[0028] Step 3.1.3, use the weight correspondence calculation in step 3.1.2 to obtain the weighted component;

[0029] In step 3.1.4, the two weighted components are concatenated and convolved before output.

[0030] Preferably, step 3 further includes the following adjustment and fusion steps for the self-attention branch:

[0031] Step 3.2.1, performing a scaling operation on the vector Q;

[0032] Step 3.2.2, performing a rearrangement operation on the vector K;

[0033] Step 3.2.3, the scaled vector Q, the rearranged vector K and the vector V are then position-adjusted and output.

[0034] Preferably, the setting rule of the image category confidence in step 4 is that the confidence of unstable objects and dynamic objects is set to 0.1; the confidence of short-term objects is set to 0.5; and the confidence of long-term objects is set to 1.

[0035] Preferably, the total loss function L described in step 5 is desc Expressed as,

[0036] L desc =L Trip +L in +L bet ;

[0037] Among them, L Trip Expressed as a semantically weighted triplet loss function, L in and L bet are the intra-class and inter-class loss functions, respectively.

[0038] Preferably, the semantically weighted triplet loss function L Trip Expressed as

[0039]

[0040] Among them, d p is the matching descriptor, d n For unmatched descriptors, d i is the descriptor, w p and w n denote the weights of the matched descriptor and the unmatched descriptor, w i is the corresponding descriptor d i The network weight score of

[0041] The intra-class loss function L in Expressed as

[0042]

[0043] Among them, dis + Expressed as the Euclidean distance between positive samples of the same class, dis - Expressed as the negative sample distance under different category labels;

[0044] The inter-class loss function L bet Expressed as

[0045]

[0046] Among them, N c is the total number of all segmentation categories, N d For C k The total number of descriptors extracted from For label C k The descriptor of Assign a reliability score to it.

[0047] As a second aspect of the present application, the present invention further discloses an electronic device, comprising:

[0048] at least one processor, and a memory communicatively coupled to the at least one processor;

[0049] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the above-mentioned local feature detection and description method based on multi-scale semantic feature fusion.

[0050] As a third aspect of the present application, the present invention further discloses a computer storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned local feature detection and description method based on multi-scale semantic feature fusion are implemented.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention provides a local feature detection and description method based on multi-scale semantic feature fusion, which can better solve the problem of weak texture repeated erroneous descriptor generation. First, the image to be processed is obtained for homography transformation, and then input into the multi-scale backbone network to extract the detection head, descriptor head, weight head and confidence head. Then, convolution and self-attention are used in parallel to refine the descriptor head to generate a descriptor. The image category confidence is set in combination with prior semantic information to guide the neglect of features on unstable objects. Finally, a descriptor total loss function is introduced to guide the generation of the confidence head. The present invention improves the adaptability of the feature extraction network to illumination and perspective changes by homography transformation of the image. The convolution and self-attention mechanisms are innovatively adopted in parallel, combined with a coarse-to-fine descriptor generation method. The prior semantic information is used to implicitly guide the network to focus on stable areas of the image and eliminate unstable elements, thereby obtaining a robust and rich feature representation. It effectively overcomes the problem of generating sparse feature descriptors under illumination and perspective changes, weak textures and repeated textures, and shows broad application prospects in related visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings that constitute part of this application are used to provide a further understanding of this application and make other features, objects and advantages of this application more apparent. The illustrative embodiment drawings of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. In the drawings:

[0054] Figure 1 A diagram showing the steps of a method for local feature detection according to an embodiment of the present invention;

[0055] Figure 2 Flowchart of a method for local feature detection according to an embodiment of the present invention;

[0056] Figure 3 for Figure 2 The detailed flow chart of the parallel structure of the descriptor is generated in the refinement;

[0057] Figure 4 for Figure 3 Detailed flowchart of processing channel attention and spatial attention in the convolution branch;

[0058] Figure 5 This is a visual comparison chart of the matching effects of different methods in the Aachen scenario in an embodiment of the present invention;

[0059] Figure 6 This is a visualization diagram of the descriptor effect on the Hpatches and Aachen scenes in an embodiment of the present invention;

[0060] Figure 7 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0062] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0063] The present invention discloses a local feature detection and description method based on multi-scale semantic feature fusion, which will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0064] Reference Figure 1 and Figure 2 , including the following steps:

[0065] Step 1: Obtain the image to be processed and perform homography transformation on the image;

[0066] Step 2: Input the homography-transformed image into the multi-scale backbone network to extract the detection head, descriptor head, weight head, and confidence head;

[0067] Step 3: Use convolution and self-attention to refine the descriptor head in parallel to generate the descriptor;

[0068] Step 4: Combine prior semantic information to set the image category confidence and guide the neglect of features on unstable objects;

[0069] Step 5: Introduce the descriptor total loss function to guide the confidence head generation;

[0070] Using a homography transformation matrix, we randomly transform images from the MegaDepth dataset, selecting images with a richer range of categories. This is then passed through a first-layer convolutional structure and a three-layer residual structure composed of four intermediate features, followed by upsampling to the original image size. The first-layer features are dynamically weighted to output a confidence header, which is then averaged through a convolution operation to sequentially output a weight header and a confidence header, directly generating a coarse-grained descriptor head. The third intermediate layer features are then refined through a parallel structure of convolution and self-attention to generate descriptors. Finally, semantic encoding and decoding of the original image is performed based on prior semantics, and confidence levels are set to guide feature point generation.

[0071] Specifically, as described in step 1, the image to be processed is acquired and a homography transformation is performed on the image. A homography transformation is typically represented by a 3×3 matrix H, which is used to map corresponding points on a uniform physical plane at different viewing angles. It is suitable for projective transformations between image planes, that is, mapping the plane points of the original image to the target plane. Given a point (x1, y1) in the acquired image to be processed, the corresponding point (x2, y2) after the homography transformation can be transformed using the following equation:

[0072]

[0073] Among them, the homography matrix H contains a composite matrix of transformations such as translation, rotation, scaling, and shearing, which respectively represent the relationship between images under different perspectives. The input original image is represented as I∈R H×W×3 , the random image corresponding to the output after homography transformation is expressed as I'∈R H×W×3 A homography represents the geometric transformation relationship between two images, mapping the original image to another image through a perspective transformation. This is intended to improve the ability of subsequent multi-scale backbone networks to handle sparse feature points with varying perspectives and illumination intensity.

[0074] Next, we input the homography-transformed image into the multi-scale backbone network to extract the detection head, descriptor head, weight head, and confidence head. The specific steps include:

[0075] Step 2.1: The image after homography transformation is mapped through a multi-scale backbone network to obtain four intermediate layer features;

[0076] Step 2.2: Fuse the last three intermediate layer features with the first intermediate layer feature in the channel dimension.

[0077] In step 2.3, the fused feature map is output to the detection head, descriptor head, weight head, and confidence head.

[0078] Specifically, the homography transformed image is input into the multi-scale backbone network for multi-layer mapping, which can capture multi-scale information using feature maps at different levels to improve the model's adaptability to target size, position and complex scenes. The homography transformed image is input into the multi-scale backbone network and mapped to obtain different features at different depth levels, thus forming a multi-scale pyramid structure represented as F1∈R H×W×32 、 as well as These four intermediate layer features. Among them, F1 is the first intermediate layer feature and uses two 3×3conv convolution structures, and F2, F3 and F4 use ResBlock structure. The ResBlock structure includes a convolution layer and a jump connection layer, and F2, F3 and F4 are similar to the last three layers of the SFD2 backbone network. The last three layers of the SFD2 backbone network usually follow the traditional three-layer network architecture model. Then, in order to integrate feature maps of different scales, we change the three intermediate layer features of F2, F3 and F4 into R by upsampling. H×W×32 And fused with F1 in the channel dimension, the feature map restored to the original image size is represented as F∈R H×W×128 . Use this feature map to generate detection heads Descriptor header Weight Head and confidence head The detection head is used to decode the location and category information of the target from the feature map. The descriptor head is used to generate instance-level feature embedding to complete target tracking, retrieval and other related tasks. The weight head is used to dynamically adjust the weight of the feature map or prediction results. The confidence head is used to evaluate the reliability of the prediction box.

[0079] In addition, during the feature fusion process, features at different levels should be given different attention levels, and a weighted feature fusion strategy should be adopted, that is, different weights are assigned to F1, F2, F3, and F4. This is because deeper features contain more high-dimensional information and global information, so the weight values ​​are higher. In order to allow the network to adaptively adjust the subtle weight distribution, the present invention adopts a dynamic weight method. Specifically, the following steps are included:

[0080] Step 2.2.1, stack the four intermediate layer features on the channel to obtain the feature map;

[0081] Step 2.2.2: linearly transform the feature map through the attention mechanism to generate the corresponding query Q, key K and value V;

[0082] Step 2.2.3, dynamically calculate the attention weight using query Q, key K and value V;

[0083] Step 2.2.4, perform weighted summation of feature maps using attention weights;

[0084] In step 2.2.5, the weighted summed feature map is used to generate a heat map observation.

[0085] Specifically, the above four intermediate layer features are stacked on the channel to obtain a feature map whose total dimension is expressed as X H ×W×C×4, where H represents the height of the feature map, W represents the width of the feature map, C represents the number of channels, and C = 32. The feature map is linearly transformed through the attention mechanism to generate the corresponding query Q, key K and value V, which are respectively represented as query Q H ×W×C×4 , key K H×W×C×4 , value V H×W×C×4 . The attention weight is then calculated as in, It is expressed as a scaling factor, and a dynamic weight is generated by calculating the similarity by the dot product of query Q and key K. The feature map is then weighted and summed using the attention weight to aggregate relevant information and suppress irrelevant information. Finally, the weighted sum feature map is fed into a 3×3 convolutional network to generate a heat map represented as The heat map is a visualization tool. Since the heat map is the external manifestation of dynamic weights, it can directly reflect the effect of dynamic weights in the feature map.

[0086] However, the descriptor head generated in step 2 is rough, and we need to further use convolution and self-attention to perform split fusion and permutation adjustment in parallel to refine the descriptor generation, such as Figure 3 As shown in Figure 2. Three 1×1 convolution kernels are used to uniformly generate vectors Q, K, and V, which serve as the common pre-order operations for the subsequent convolution branch and the self-attention branch. Figure 4 As shown in the figure, the split fusion steps for the convolution branch are as follows:

[0087] Step 3.1.1, concatenate vectors Q, K, and V to obtain the fused vector;

[0088] Step 3.1.2: Generate corresponding weights by fusing the vectors through the channel attention branch and the spatial attention branch.

[0089] Step 3.1.3, use the weight correspondence calculation in step 3.1.2 to obtain the weighted component;

[0090] In step 3.1.4, the two weighted components are concatenated and convolved before output.

[0091] First, vectors Q, K, and V are concatenated on the channel, which is performed using the cat function. The concatenated dimension is the sum of the dimensions of the three vectors, and then passed through the fully connected layer f c Get the fusion vector, expressed as F cov =f c (cat(Q, K, V)), which can achieve local feature enhancement and cross-attention interaction. The vector is further passed through the channel attention branch side and the spatial attention side to generate the corresponding weight W c and W sThe channel attention branch x0 first uses the global average pooling (Avg), and then performs weighted (Sigmoid) calculation on the channel to obtain the channel weighted component x0 expressed as,

[0092] x0=Sigmoid(W c Avg(x0)+b c )·x0

[0093]

[0094] Among them, W c and b c They are respectively represented as the weight and bias of the channel attention branch side, and both are learnable parameters. For the spatial attention branch x1, it is first normalized, and then the spatial weight is calculated to obtain the spatial weighted component x1, which is expressed as,

[0095] x1=Sigmoid(W s GN(x1)+b s ) x1

[0096]

[0097] Among them, W s and b s They are respectively represented as the weight and bias of the spatial attention branch. Then the channel weighted component x0 and the spatial weighted component x1 are concatenated and restored to their original size (H×W×128) after the convolution operation. The output of the convolution branch is represented as

[0098] In addition, the permutation adjustment steps in the self-attention branch include:

[0099] Step 3.2.1, scaling the vector Q;

[0100] Step 3.2.2, rearrange the vector K;

[0101] In step 3.2.3, the scaled vector Q, rearranged vector K, and vector V are repositioned and then output.

[0102] The query Q and key K generated above are rearranged and scaled to be expressed as:

[0103]

[0104] Among them, H dRepresents the dimension of each attention head. Unfold refers to extracting image patches by sliding from left to right and from top to bottom through a window of a specific size and stride. Pad_att refers to the reflection padding operation, which expands the input size by reflecting image pixels at the edge of the input to avoid the loss of edge information. Then, the output of the self-attention branch is expressed as,

[0105]

[0106] Among them, PE stands for Positional Encoding, and RPE stands for Relative Positional Encoding. The final output feature aggregation is expressed as the learnable weight aggregation

[0107] Step 3 reduces the size of the self-attention mechanism by replacing the 3x3 convolution kernel with a 1x1 convolution kernel. Both the convolution branch and the self-attention branch use the same first stage: 1x1 convolutions for feature extraction. The intermediate features generated by the three 1x1 convolutions are actually reshaped to produce three sets of features, each consisting of feature branches corresponding to the number of attention heads. Since the convolution branch uses a 1x1 convolution kernel instead of a 3x3 convolution kernel, its corresponding expanded component is actually a feature map of the square of the 9 convolution kernel sizes. A fully connected layer reduces the third dimension from 12 to 9, and these 9 feature submaps are then split and merged to further extract more information across channels and space. Finally, permute and reshape operations are performed to restore the original state. The attention branch performs self-attention on each head before concatenating them.

[0108] By splitting channel and spatial attention in the convolution branch, the significance of this design is that the channel attention mechanism focuses on the different channel dimensions of the input feature map. Its purpose is to allow the model to dynamically adjust the weight of each channel according to the importance of different feature map channels of the input. The channel attention mechanism focuses on each channel, that is, the information processing within the depth of the input feature map, without considering the spatial position. The spatial attention mechanism focuses on different spatial positions in the feature map (that is, the position on the two-dimensional plane), and its purpose is to allow the model to dynamically adjust the response of the feature map according to the importance of the spatial position. The spatial attention mechanism focuses on different positions in the two-dimensional space, helping the model to focus on important areas. However, both still focus on local information acquisition. To this end, by combining the global mechanism of the self-attention mechanism, global contextual information is captured to form a global dependency relationship. The combination of the two forms information acquisition from regional to global, from single layer to multi-dimensional.

[0109] As described in step 4, prior semantic information is introduced into prior image recognition, and confidence values ​​for different image categories are set. Using the SegNet network architecture, prior image recognition is performed on 150 pre-defined categories, with mask values ​​set to four manually defined category confidence values. Prior semantic information, defined by semantic constraints or rules based on domain knowledge, filters out unnecessary features during semantic segmentation, thereby constraining the generation of feature points and descriptors in the loss function. Semantic segmentation labels are categorized into four categories: unstable objects, dynamic objects, long-term objects, and short-term objects. Feature points in areas with repetitive textures, such as the sky and water, are unreliable. Moving objects, such as pedestrians and vehicles, or objects with potential motion, can lead to mismatched feature points and should be eliminated. Short-term objects, such as trees, require lower confidence than long-term objects, such as landmarks and buildings. The specific confidence setting rule is that features on unstable and dynamic objects are not completely unreliable. For example, a vehicle that is temporarily static or relatively stationary has a confidence value of 0.1. Short-term objects (primarily plants) will change their appearance with the seasons, so their confidence is set to 0.5. Long-term objects remain largely unchanged, so their stability is set to 1.0 at most.

[0110] In step 5, the loss function is introduced to guide the feature descriptor to actively focus on stable features. The semantically weighted triplet loss and intra-class and inter-class cross entropy loss are used to guide the generation of confidence heads, thereby improving the implicit representation ability of descriptors and feature points. Specifically, the existing descriptor loss adopts the standard triplet loss, which forces the matching descriptor pairs to be as close as possible and the mismatching descriptor pairs to be as far as possible. The standard triplet loss is expressed as,

[0111]

[0112] Among them, d p is the matching descriptor, d n is the mismatched descriptor, and t represents the minimum Euclidean distance between the two. However, we incorporate the consistent attention mechanism into the standard triplet loss, jointly optimizing the direction of the descriptor in the optimization direction of the attention score and the angle optimization direction of the standard triplet loss, while gradually expanding the description range of the descriptor instead of directly expanding from local to global. The modified consistent attention triplet loss is expressed as,

[0113]

[0114] Among them, w i is the corresponding descriptor d i The network weight score, w p and w nDenote the weights of the matched descriptor and the unmatched descriptor respectively, and T is a smoothing factor used to adjust the influence level of the descriptor weight. Since the semantic information introduces the relevant area of ​​confidence, the intermediate semantic layer contains the flattened component of the pixel-by-pixel confidence of the original image size. Where l = H × W, as an additional product term of the network weights, the modified semantically weighted triplet loss is expressed as,

[0115]

[0116] Among them, v i The category confidence settings are slightly different. The confidence of the original dynamic objects and invalid objects is set to 0 to ensure that v i Although the above formula does not appear to exist after normalization, it ensures that the elements set to zero remain zero after normalization.

[0117] In order to better utilize the constraints of global semantic information, the principle of descriptor generation is extended to the entire image, and the corresponding intra-class loss is set according to the different segmentation results as follows:

[0118]

[0119] Among them, dis + Expressed as the Euclidean distance between positive samples of the same class, dis - It is expressed as the distance between negative samples under different category labels. The inter-class loss is expressed as the average precision loss APloss,

[0120]

[0121] Among them, N c is the total number of all segmentation categories, N d For C k The total number of descriptors extracted from For label C k The descriptor of is its corresponding reliability score. Therefore, the total descriptor loss is expressed as L desc =L Trip +L in +L bet , the overall loss of the network is expressed as L all =L desc +L det Among them, L Trip is the semantically weighted triplet loss, L in and L bet are intra-class and inter-class losses, L in and L bet The combination is expressed as intra-class and inter-class cross entropy loss. Figure 5This is a visual comparison of the matching effects of different methods in the Aachen scene. Figure 6 A visualization of the descriptor effect on the Hpatches and Aachen scenes.

[0122] In order to implement the above embodiment, the present application also discloses an electronic device. Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0123] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0124] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer storage medium, and the computer program includes program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network via the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0125] It should be noted that the computer storage medium described above in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0126] In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0127] The computer storage medium may be included in the electronic device or may exist independently without being incorporated into the electronic device. The computer storage medium carries one or more programs that, when executed by the electronic device, enable the electronic device to implement the local feature detection and description method.

[0128] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0129] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the part of the module, program segment or code includes one or more executable instructions for realizing the logical function of the specification. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings.

[0130] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0131] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A local feature detection and description method based on multi-scale semantic feature fusion, characterized in that: The steps include: Step 1: Obtain the image to be processed and perform homography transformation; Step 2: Input the homography-transformed image into the multi-scale backbone network to extract the detection head, descriptor head, weight head, and confidence head; Step 3, using convolution and self-attention to concurrently refine the descriptor head to generate a descriptor; Step 4: Combine prior semantic information to set the image category confidence and guide the neglect of features on unstable objects; Step 5: Introduce the descriptor total loss function to guide the confidence header generation.

2. The local feature detection and description method based on multi-scale semantic feature fusion according to claim 1, characterized in that: The steps of step 2 include the following steps: Step 2.1, the image after homography transformation is passed through a multi-scale backbone network through multi-layer mapping to obtain four intermediate layer features; Step 2.2: Fuse the last three intermediate layer features with the first intermediate layer feature in the channel dimension. Step 2.3: Output the detection head, descriptor head, weight head, and confidence head through the fused feature map.

3. The local feature detection and description method based on multi-scale semantic feature fusion according to claim 2, characterized in that: The step 2.2 is to use dynamic weighted feature fusion, which also includes the following steps: Step 2.2.1, stacking the four intermediate layer features on the channel to obtain a feature map; Step 2.2.2, linearly transform the feature map through the attention mechanism to generate the corresponding query Q, key K and value V; Step 2.2.3, dynamically calculate the attention weight using the query Q, key K and value V; Step 2.2.4, performing weighted summation on the feature map using the attention weights; In step 2.2.5, the weighted summed feature map is used to generate a heat map for observation.

4. The local feature detection and description method based on multi-scale semantic feature fusion according to claim 2, characterized in that: In step 3, vectors Q, K, and V are first generated by operating with three 1×1 convolution kernels, and the split-fusion steps for the convolution branches are as follows: Step 3.1.1, concatenate the vectors Q, K, and V to obtain a fusion vector; Step 3.1.2, generating corresponding weights by passing the fusion vector through the channel attention branch side and the spatial attention branch side; Step 3.1.3, using the weights in step 3.1.2 to calculate the weighted components; In step 3.1.4, the two weighted components are concatenated and convolved before output.

5. The local feature detection and description method based on multi-scale semantic feature fusion according to claim 4, characterized in that: Step 3 also includes the following adjustment and fusion steps for the self-attention branch: Step 3.2.1, performing a scaling operation on the vector Q; Step 3.2.2, performing a rearrangement operation on the vector K; Step 3.2.3, the scaled vector Q, the rearranged vector K and the vector V are then position-adjusted and output.

6. The method for local feature detection and description based on multi-scale semantic feature fusion according to claim 5, characterized in that: The setting rule of the image category confidence in step 4 is as follows: the confidence of unstable objects and dynamic objects is set to 0.1; the confidence of short-term objects is set to 0.5; The confidence of long-term objects is set to 1.

7. The method for local feature detection and description based on multi-scale semantic feature fusion according to claim 6, characterized in that: The descriptor total loss function L in step 5 desc Expressed as, L desc =L Trip +L in +L bet ; Among them, L Trip Expressed as a semantically weighted triplet loss function, L in and L bet are the intra-class and inter-class loss functions, respectively.

8. The method for local feature detection and description based on multi-scale semantic feature fusion according to claim 7, characterized in that: The semantically weighted triplet loss function L Trip Expressed as Among them, d p is the matching descriptor, d n For unmatched descriptors, d i is the descriptor, w p and w n denote the weights of the matched descriptor and the unmatched descriptor, w i is the corresponding descriptor d i The network weight score of The intra-class loss function L in Expressed as Among them, dis + Expressed as the Euclidean distance between positive samples of the same class, dis - Expressed as the distance between negative samples under different category labels; the inter-class loss function L bet Expressed as Among them, N c is the total number of all segmentation categories, N d For C k The total number of descriptors extracted from For label C k The descriptor of Assign a reliability score to it.

9. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 8.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps according to any one of claims 1 to 8 are implemented.

Images