Semantic matching method based on visual features of large-scale pre-training model

Abstract

The invention discloses a semantic matching method based on visual features of a large-scale pre-training model, and the method comprises the following steps: obtaining an image data set which comprises a target image and a reference image with corresponding semantics; constructing a corresponding semantic matching network based on the visual basic model framework, wherein the semantic matching network comprises a feature extraction module, a staggered sensing module, a matching module and an up-sampling module; training a semantic matching network by using the image data set to obtain a semantic matching model for image semantic matching; and inputting the reference image, the to-be-matched points on the reference image and the target image into a semantic matching model to output matched points on the target image. According to the method provided by the invention, the accuracy of the semantic matching task can be effectively improved, and a better service is provided for a downstream visual task.

Description

Technical Field

[0001] The present invention belongs to the field of semantic matching in computer vision, and in particular relates to a semantic matching method based on visual features of a large-scale pre-trained model. Background Art

[0002] Semantic matching is a fundamental task in computer vision and is of great significance for tasks such as image classification, small sample segmentation, video object segmentation, and object tracking. The goal of semantic correspondence is to establish sparse pixel-to-pixel correspondence between two images. In particular, semantic matching emphasizes matching different instances belonging to the same category. In contrast, other matching tasks, such as optical flow, usually focus on matching the same instance between different frames. Therefore, in addition to low-level features, semantic matching is a task that requires high-level semantic information (such as category information). In past studies, semantic matching remains an unsolved challenge due to the lack of a general visual backbone network that can provide comprehensive high-level semantic information.

[0003] A typical semantic matching model usually consists of a backbone network and a matching module. The backbone network extracts features from the image, and the matching module uses these features to calculate the similarity matrix. Previous work usually further integrates and enhances the similarity matrix, and finally extracts the matching results from the similarity matrix.

[0004] Academic literature Transformatcher: Match-to-match attention for semantic correspondence. In Proceedings of IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. discloses a semantic matching model, but ignores the fact that the quality of extracted features directly affects the matching results. Most existing semantic matching networks rely on convolutional neural networks (CNNs) pre-trained on the ImageNet dataset for image classification tasks as a backbone, and the quality of extracted features is not sufficient to directly produce satisfactory matching results. Therefore, the matching modules in previous works are designed to be complex, resulting in high computational overhead. In recent years, with the emergence of visual transformers (ViT) and various self-supervised pre-training methods, the backbone network can obtain more powerful features.

[0005] Patent document CN118333538A discloses an interface message automatic adaptation method, device, equipment and medium, the method comprising: performing semantic recognition processing on the interface messages of each supplier through a linear iterative distributed average algorithm; based on the semantic recognition result, generating a k-diagonal circulant matrix and a circulant band matrix, performing a first semantic matching process based on the k-diagonal circulant matrix and the circulant band matrix, and obtaining a first semantic matching matrix; based on the semantic recognition result and a diffusion algorithm, calculating a diffusion similarity matrix, performing a second semantic matching process based on the diffusion similarity matrix, and obtaining a second semantic matching matrix; based on the first semantic matching matrix and the second semantic matching matrix, performing cluster analysis on the interface messages of different suppliers, and outputting the adapted standard data corresponding to the interface messages of each supplier. Summary of the invention

[0006] The purpose of the present invention is to provide a semantic matching method based on the visual features of a large-scale pre-trained model, which can effectively improve the accuracy of semantic matching tasks and provide better services for downstream visual tasks.

[0007] In order to achieve the purpose of the present invention, the following technical solution is provided: a semantic matching method based on visual features of a large-scale pre-trained model, comprising the following steps:

[0008] Acquire an image dataset, which includes a target image and a reference image corresponding to the semantics;

[0009] Building a corresponding semantic matching network based on the visual basic model framework, wherein the semantic matching network includes a feature extraction module, an interlaced perception module, a matching module and an upsampling module;

[0010] The feature extraction module includes a common image encoder and a high-resolution image encoder, wherein the common image encoder is used to perform feature extraction on the input target image and the reference image to obtain corresponding target image features and reference image features, and the high-resolution image encoder is used to perform high-resolution feature extraction on the input target image and the reference image to obtain corresponding high-resolution reference image features and high-resolution target image features;

[0011] The interlaced perception module is used to perform image enhancement on the target image features and the reference image features to output corresponding enhanced reference image features and enhanced target image features;

[0012] The matching module is used to calculate the semantic similarity between the enhanced reference image features and the enhanced target image features to construct a corresponding semantic similarity matrix, and calculate the corresponding initial matching flow based on the semantic similarity matrix, and calculate the similarity between the high-resolution reference image features and the high-resolution target image features to construct a corresponding high-resolution texture similarity matrix;

[0013] The upsampling module uses the initial matching stream to crop the constructed high-resolution texture similarity matrix to output a local texture similarity matrix, concatenates and upsamples the local texture similarity matrix with the initial matching stream, and performs a convolution regression operation on the upsampling result to output a predicted matching result of the target image;

[0014] The semantic matching network is trained using the image dataset to obtain a semantic matching model for image semantic matching;

[0015] The reference image, the points to be matched on the reference image, and the target image are input into the semantic matching model to output the matching points on the target image.

[0016] The present invention adopts the staggered attention operation to perform feature enhancement on the target image and the reference image to enhance their visual expression ability, and further refines the final matching result through the upsampling matching process guided by the auxiliary loss function and the high-resolution similarity matrix.

[0017] Specifically, the interlaced perception module uses an interlaced attention operation to cross-enhance the target image features and the reference image features, thereby enhancing their visual expression capabilities and obtaining reliable matching results. The specific operations are as follows:

[0018] The enhanced reference image feature is obtained by enhancing the reference image feature using the target image feature as the material;

[0019] The enhanced target image feature is obtained by enhancing the target image feature using the reference image feature as a material.

[0020] Specifically, the interleaved perception module includes two attention blocks, and each attention block is provided with a cross attention layer, a self-attention layer and a feedforward network in sequence along the image feature input direction.

[0021] Specifically, the expression of the staggered attention operation is as follows:

[0022] F A′ ←FFN(SelfAttn(CrossAttn(F A ,F B )))

[0023] F B′ ←FFN(SelfAttn(CrossAttn(F B ,F A )))

[0024] Among them, F A represents the reference image feature, F Brepresents the target image features, F A′ represents the enhanced reference image features, F B′ Represents the enhanced target image features.

[0025] Specifically, the expression of the semantic similarity matrix is ​​as follows:

[0026]

[0027] Among them, F A′ represents the enhanced reference image features, F B′ Represents the enhanced target image features.

[0028] Specifically, the calculation process of the initial matching flow is as follows:

[0029] A 2D Gaussian kernel is applied to the semantic similarity matrix to suppress non-maximal local maxima, and the semantic similarity matrix is ​​normalized along the last two dimensions to obtain the corresponding correlation distribution;

[0030] The last two dimensions of the semantic similarity matrix are used as weights and weighted summed in a two-dimensional coordinate grid to obtain densely corresponding key point predictions;

[0031] The initial matching flow is obtained by calculating using the following formula:

[0032]

[0033]

[0034] Among them, P represents the relevant distribution, G represents the densely corresponding key point prediction, and M represents the initial matching flow.

[0035] Specifically, the expression of the local texture similarity matrix is ​​as follows:

[0036]

[0037] Where p = (x, y) represents the coordinates of the target pixel in the reference image feature, Δp∈M is the initial predicted matching flow of the target pixel, represents the local neighborhood of a pixel, and r is the neighborhood radius.

[0038] Specifically, during the training process, the matching network is trained using a joint loss function consisting of endpoint error loss and auxiliary triplet loss to back-propagate and optimize the weight parameters of the matching network, which is expressed as follows:

[0039]

[0040] in, represents the auxiliary triplet loss, represents the endpoint error loss, and λ1 and λ2 represent the weights.

[0041] Specifically, the auxiliary triplet loss takes the features of the key points in the image as samples for constructing triplet, i.e., p = {p1, p2, ..., p N} and q={q1,q2,...,q N} are the coordinates of the key points in the reference image and the target image, respectively, to construct the corresponding loss calculation formula:

[0042]

[0043] in, Represents the feature of the location of the i-th key point in the reference image, Represents the feature of the location of the i-th key point in the target image, It represents the features of the locations of non-corresponding key points in the target image, and N represents the total number of samples.

[0044] Specifically, the expression of the endpoint error loss is as follows:

[0045]

[0046] in, and are the real matching flow and the predicted matching flow respectively, and N represents the total number of samples.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The interlaced perception module effectively strengthens the joint modeling of two images by the visual basic model, enhances its visual expression ability, and thus obtains reliable matching results;

[0049] The matching results are further refined by using an auxiliary loss function and an upsampling module guided by a high-resolution similarity matrix. The auxiliary loss function is mainly used to distinguish objects with similar semantics, while the upsampling module guided by a high-resolution similarity matrix uses fine-grained information to make positioning more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a semantic matching method based on large-scale pre-trained model visual features provided in this embodiment;

[0051] Figure 2 A schematic diagram of the flow of the semantic matching network provided in this embodiment;

[0052] Figure 3 A schematic diagram of the flow of the interlaced perception module provided in this embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0054] like Figure 1 As shown, a semantic matching method is provided in this embodiment. The method optimizes the visual backbone network DINOv2 by designing a matching module to maximize the potential of the network in semantic matching tasks, obtain more accurate matching results, and better serve downstream visual tasks.

[0055] The specific steps are as follows:

[0056] An image dataset is obtained, which includes a target image and a reference image of corresponding semantics.

[0057] Based on the visual basic model framework, a corresponding semantic matching network is constructed, which includes a feature extraction module, an interlaced perception module, a matching module and an upsampling module. Figure 2 The figure is a schematic diagram of the flow of the semantic matching network provided by this embodiment.

[0058] The feature extraction module includes a common image encoder and a high-resolution image encoder, wherein the common image encoder is used to perform feature extraction on the input target image and the reference image to obtain corresponding target image features and reference image features, and the high-resolution image encoder is used to perform high-resolution feature extraction on the input target image and the reference image to obtain corresponding high-resolution reference image features and high-resolution target image features;

[0059] The interlaced perception module is used to perform image enhancement on the target image features and the reference image features to output corresponding enhanced reference image features and enhanced target image features;

[0060] The matching module is used to calculate the semantic similarity between the enhanced reference image features and the enhanced target image features to construct a corresponding semantic similarity matrix, and calculate the corresponding initial matching flow based on the semantic similarity matrix, and calculate the similarity between the high-resolution reference image features and the high-resolution target image features to construct a corresponding high-resolution texture similarity matrix;

[0061] The upsampling module uses the initial matching flow to crop the constructed high-resolution texture similarity matrix to output a local texture similarity matrix, concatenates and upsamples the local texture similarity matrix with the initial matching flow, and performs a convolution regression operation on the upsampling result to output a predicted matching result of the target image.

[0062] The feature extraction module includes a common image encoder and a high-resolution image encoder. The working principle of each encoder is as follows: given a reference key point K A Reference image of the location and a target image The network is designed to predict the target keypoint location K B , with the reference key point K A Correspondingly, for the target image I B . K A It can be dense or sparse, depending on the task requirements. The visual backbone network first starts with two input images I A and I B Extract feature maps C represents the output dimension of the visual backbone; S represents the step size of the feature map. In this embodiment, all feature maps F A and F B All are upsampled to a step size of S=8.

[0063] like Figure 3 As shown in FIG. 1 , the interlaced perception module provided by this embodiment can significantly enhance the feature expression capability by only a single layer. The input of the interlaced perception module is the feature map F A and F B These feature maps are passed through two attention blocks in turn. Each block consists of a criss-cross attention layer, a self-attention layer, and a feed-forward network in sequence. The criss-cross attention layer is used to fuse features across frames, while the self-attention layer is used to update features within each frame. Different from previous methods, we find that alternating updates of F A and F B can improve the performance. Specifically, in the first attention block of the interleaved attention module, F A As query, F B As key and value, F BThe features of F were initially integrated into A′ In contrast, in the subsequent second attention block, F B As a query, and F A As key and value, make F A The features of F are fused into B′ The process can be expressed by the following formula:

[0064] F A′ ←FFN(SelfAttn(CrossAttn(F A ,F B )))

[0065] F B′ ←FFN(SelfAttn(CrossAttn(F B ,F A )))

[0066] Among them, F A represents the reference image feature, F B represents the target image features, F A′ represents the enhanced reference image features, F B′ Represents the enhanced target image features.

[0067] After the interleaved perception module completes feature enhancement, the matching module calculates the semantic similarity matrix based on cosine similarity for the enhanced features to construct the corresponding semantic similarity matrix.

[0068] The specific calculation method is the semantic similarity matrix in As shown below:

[0069]

[0070] C contains the enhanced reference image features F A All pixels in the image are compared with the enhanced target image feature F B The correlation score between all pixels in .

[0071] The prediction keypoints are extracted from the correlation matrix C using the following soft-argmax procedure. First, a two-dimensional Gaussian kernel is applied to C to suppress non-maximal local maxima. Then, C is normalized along the last two dimensions to obtain the correlation distribution

[0072] P = Softmax(C).

[0073] Then, using the last two dimensions of C as weights, we can create a grid of Using weighted summation, we can get dense corresponding key point predictions

[0074]

[0075] It is worth noting that the temperature coefficient is introduced to prevent the correlation matrix from being over-smoothed. In this model, the temperature is configured as a learnable parameter that can be optimized during the training process. Then, the matching flow graph can be obtained by the following method

[0076]

[0077] After upsampling the matching flow graph to the resolution of the original image, dense correspondence is obtained from each pixel in the source image to the target image. The key points K of the sparse prediction are extracted from B It can be extracted from P according to the index.

[0078] In this embodiment, a high-resolution texture similarity matrix is ​​used to guide the upsampling operation. In the specific process, a specific local similarity matrix is ​​used instead of a global similarity matrix. This choice is because long-range similarity is unnecessary for upsampling and may have a negative impact on the results because the high-resolution texture similarity matrix is ​​generated by a shallow network and only contains local information. Local similarity contains the relationship information between the source pixel and the surrounding pixels that may correspond. Then, the matching flow graph and local similarity are upsampled to the full resolution through bilinear interpolation and encoded by their respective convolutional neural networks. Finally, they are connected and input into another convolutional neural network for encoding to obtain the final predicted matching flow at the full resolution.

[0079] The semantic matching network is trained using the image dataset to obtain a semantic matching model for image semantic matching.

[0080] In this embodiment, a joint loss function consisting of endpoint error loss and auxiliary triplet loss is used for training. The endpoint error loss is defined according to the following formula:

[0081]

[0082] in and They are the real matching flow and the predicted matching flow respectively.

[0083] The auxiliary triplet loss is defined as follows. The general triplet loss is defined as

[0084] L(a,p,n)=max(d(a,p)-d(a,n)+m,0),

[0085] Where a, p and n are the anchor point, positive sample and negative sample of the triplet respectively, d is the distance function (such as L2 distance), and m is the margin. In the present invention, we use the features of the key point location as the sample to construct the triplet. Let p = {p1, p2, ..., p N} and q={q1,q2,...,q N} are the coordinates of the key points of the source image and the target image respectively, then the auxiliary function can be defined as

[0086]

[0087] in and They are the features of a key point in the source image, the features of the corresponding key point in the target image, and the features of the non-corresponding key point in the target image. In training, we use weights λ1 and λ2 to balance the two losses, defined as

[0088]

[0089] After obtaining the loss, the model parameters can be optimized through back propagation.

[0090] The target image to be semantically matched is input into the semantic matching model to output a reference image that semantically matches the specified image database to better serve downstream visual tasks.

[0091] In addition, the terms "upper", "lower", "inner", "outer", "front", "rear" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps described in these embodiments do not limit the scope of the present invention.

[0092] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of implementation of the present invention. All equivalent changes or modifications made according to the structure, characteristics and principles described in the patent application scope of the present invention should be included in the patent application scope of the present invention.

[0093] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A semantic matching method based on visual features of a large-scale pre-trained model, characterized in that: The following steps are involved: Acquire an image dataset, which includes a target image and a reference image corresponding to the semantics; Building a corresponding semantic matching network based on the visual basic model framework, wherein the semantic matching network includes a feature extraction module, an interlaced perception module, a matching module and an upsampling module; The feature extraction module includes a common image encoder and a high-resolution image encoder, wherein the common image encoder is used to perform feature extraction on the input target image and the reference image to obtain corresponding target image features and reference image features, and the high-resolution image encoder is used to perform high-resolution feature extraction on the input target image and the reference image to obtain corresponding high-resolution reference image features and high-resolution target image features; The interlaced perception module is used to perform image enhancement on the target image features and the reference image features to output corresponding enhanced reference image features and enhanced target image features; The matching module is used to calculate the semantic similarity between the enhanced reference image features and the enhanced target image features to construct a corresponding semantic similarity matrix, and calculate the corresponding initial matching flow based on the semantic similarity matrix, and calculate the similarity between the high-resolution reference image features and the high-resolution target image features to construct a corresponding high-resolution texture similarity matrix; The upsampling module uses the initial matching stream to crop the constructed high-resolution texture similarity matrix to output a local texture similarity matrix, concatenates and upsamples the local texture similarity matrix with the initial matching stream, and performs a convolution regression operation on the upsampling result to output a predicted matching result of the target image; The semantic matching network is trained using the image dataset to obtain a semantic matching model for image semantic matching; The reference image, the points to be matched on the reference image, and the target image are input into the semantic matching model to output the matching points on the target image.

2. The semantic matching method based on large-scale pre-trained model visual features according to claim 1 is characterized in that: The interlaced perception module uses an interlaced attention operation to cross-enhance the target image features and the reference image features, and its specific operation is as follows: The enhanced reference image feature is obtained by enhancing the reference image feature using the target image feature as the material; The enhanced target image feature is obtained by enhancing the target image feature using the reference image feature as a material.

3. The semantic matching method based on large-scale pre-trained model visual features according to claim 1 or 2, characterized in that: The interlaced perception module includes two attention blocks, each of which is provided with a cross attention layer, a self-attention layer and a feedforward network in sequence along the image feature input direction.

4. The semantic matching method based on large-scale pre-trained model visual features according to claim 2 is characterized in that: The expression of the staggered attention operation is as follows: Among them, F A represents the reference image feature, F B represents the target image features, F A′ represents the enhanced reference image feature, F B′ Represents the enhanced target image features.

5. The semantic matching method based on large-scale pre-trained model visual features according to claim 1, characterized in that: The expression of the semantic similarity matrix is ​​as follows: Among them, F A′ represents the enhanced reference image feature, F B′ Represents the enhanced target image features.

6. The semantic matching method based on large-scale pre-trained model visual features according to claim 1, characterized in that: The calculation process of the initial matching flow is as follows: A 2D Gaussian kernel is applied to the semantic similarity matrix to suppress non-maximal local maxima, and the semantic similarity matrix is ​​normalized along the last two dimensions to obtain the corresponding correlation distribution; The last two dimensions of the semantic similarity matrix are used as weights and weighted summed in a two-dimensional coordinate grid to obtain densely corresponding key point predictions; The initial matching flow is obtained by calculating using the following formula: Among them, P represents the relevant distribution, G represents the densely corresponding key point prediction, and M represents the initial matching flow.

7. The semantic matching method based on large-scale pre-trained model visual features according to claim 1, characterized in that: The expression of the local texture similarity matrix is ​​as follows: Where p = (x, y) represents the coordinates of the target pixel in the reference image feature, Δp∈M is the initial predicted matching flow of the target pixel, represents the local neighborhood of a pixel, and r is the neighborhood radius.

8. The semantic matching method based on large-scale pre-trained model visual features according to claim 1, characterized in that: During the training process, the matching network is trained using a joint loss function consisting of endpoint error loss and auxiliary triplet loss to back-propagate and optimize the weight parameters of the matching network, which is expressed as follows: in, represents the auxiliary triplet loss, represents the endpoint error loss, λ1 and p2 represent the weights.

9. The semantic matching method based on large-scale pre-trained model visual features according to claim 8, characterized in that: The auxiliary triplet loss takes the features of the key points in the image as samples to construct triplet, i.e., p = {p1, p2, ..., p N } and q={q1,q2,...,q N } are the coordinates of the key points in the reference image and the target image, respectively, to construct the corresponding loss calculation formula: in, Represents the feature of the location of the i-th key point in the reference image, Represents the feature of the location of the i-th key point in the target image, It represents the features of the locations of non-corresponding key points in the target image, and N represents the total number of samples.

10. The semantic matching method based on large-scale pre-trained model visual features according to claim 8, characterized in that: The expression of the endpoint error loss is as follows: in, and are the real matching flow and the predicted matching flow respectively, and N represents the total number of samples.

Images