A semantic matching method based on a large-scale pre-training model visual feature

Abstract

The application discloses a semantic matching method based on a large-scale pre-training model visual feature, comprising the following steps: acquiring an image data set, which comprises a target image and a reference image corresponding to a semantic; constructing a corresponding semantic matching network based on a visual basic model framework, wherein the semantic matching network comprises a feature extraction module, an interlaced perception module, a matching module and an up-sampling module; training the semantic matching network by using the image data set to obtain a semantic matching model used for image semantic matching; and inputting the reference image, a point to be matched on the reference image and the target image into the semantic matching model to output a matched point on the target image. The method provided by the application can effectively improve the accuracy of a semantic matching task and provide better services for downstream visual tasks.

Description

Technical Field

[0001] The present invention belongs to the field of semantic matching in computer vision, and specifically relates to a semantic matching method based on visual features of a large-scale pre-trained model. Background Art

[0002] Semantic matching is a fundamental task in computer vision and is of great significance for tasks such as image classification, small sample segmentation, video object segmentation, and object tracking. The goal of semantic correspondence is to establish sparse pixel-to-pixel correspondence between two images. In particular, semantic matching emphasizes matching different instances belonging to the same category. In contrast, other matching tasks, such as optical flow, usually focus on matching the same instance between different frames. Therefore, in addition to low-level features, semantic matching is a task that requires high-level semantic information (such as category information). In past studies, semantic matching remains an unsolved challenge due to the lack of a general visual backbone network that can provide comprehensive high-level semantic information.

[0003] A typical semantic matching model consists of a backbone network and a matching module. The backbone network extracts features from the image, and the matching module uses these features to calculate a similarity matrix. Previous work typically further integrates and enhances the similarity matrix, ultimately extracting matching results from the similarity matrix.

[0004] The academic paper Transformatcher: Match-to-match attention for semantic correspondence. In Proceedings of IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. discloses a semantic matching model, but ignores the fact that the quality of the extracted features directly affects the matching results. Most existing semantic matching networks rely on convolutional neural networks (CNNs) pre-trained on the ImageNet dataset for image classification tasks as a backbone, and the quality of the extracted features is not sufficient to directly produce satisfactory matching results. Therefore, the matching modules in previous works are designed to be complex, resulting in high computational overhead. In recent years, with the emergence of visual transformers (ViT) and various self-supervised pre-training methods, the backbone network can obtain more powerful features.

[0005] Patent document CN118333538A discloses an interface message automatic adaptation method, device, equipment and medium, the method comprises: through linear iterative distributed average algorithm, the interface message of each supplier is carried out semantic recognition processing;Based on the semantic recognition result, generate k-diagonal circulant matrix and cyclic band matrix, based on k-diagonal circulant matrix and cyclic band matrix, first semantic matching processing is carried out, and first semantic matching matrix is obtained;Based on the semantic recognition result and diffusion algorithm, the diffusion similarity matrix is calculated, and the second semantic matching processing is carried out based on the diffusion similarity matrix, and the second semantic matching matrix is obtained;Based on the first semantic matching matrix and the second semantic matching matrix, the interface message of different suppliers is carried out cluster analysis, and the interface message corresponding to the standard data of each supplier after adaptation is output. SUMMARY

[0006] The purpose of the present application is to provide a semantic matching method based on large-scale pre-training model visual features, which can effectively improve the accuracy of semantic matching tasks and provide better services for downstream visual tasks.

[0007] In order to achieve the purpose of the present application, the following technical scheme is provided: a semantic matching method based on large-scale pre-training model visual features, comprising the following steps:

[0008] Obtain an image data set, which includes a target image and a reference image corresponding to the semantics;

[0009] Based on the visual basic model framework, a corresponding semantic matching network is constructed, which includes a feature extraction module, an interleaved perception module, a matching module and an up-sampling module;

[0010] The feature extraction module includes a normal image encoder and a high-resolution image encoder, the normal image encoder is used for feature extraction of the input target image and reference image to obtain corresponding target image features and reference image features, and the high-resolution image encoder is used for high-resolution feature extraction of the input target image and reference image to obtain corresponding high-resolution reference image features and high-resolution target image features;

[0011] The interleaved perception module is used for image enhancement of the target image features and the reference image features to output corresponding enhanced reference image features and enhanced target image features;

[0012] The matching module is used for calculating the semantic similarity between the enhanced reference image features and the enhanced target image features to construct a corresponding semantic similarity matrix, and calculating the initial matching flow based on the semantic similarity matrix, and calculating the similarity of the features between the high-resolution reference image features and the high-resolution target image features to construct a corresponding high-resolution texture similarity matrix.

[0013] The upsampling module crops the constructed high-resolution texture similarity matrix with the initial matching stream to output a local texture similarity matrix, splices and upsamples the local texture similarity matrix with the initial matching stream, and performs a convolution regression operation on the upsampled result to output a predicted matching result of the target image.

[0014] The semantic matching network is trained using an image dataset to obtain a semantic matching model for image semantic matching.

[0015] The reference image and the point to be matched on the reference image, and the target image are input into the semantic matching model to output a matching point on the target image.

[0016] The present application adopts the interleaved attention operation to enhance the features of the target image and the reference image, to enhance their visual expression ability, and at the same time, through the auxiliary loss function and the high-resolution similarity matrix guided upsampling matching process, to further refine the final matching result.

[0017] Specifically, the interleaved perception module adopts interleaved attention operation to cross-enhance the target image features and the reference image features, thereby enhancing their visual expression ability, and further obtaining reliable matching results, and the specific operation is as follows:

[0018] The enhanced reference image features are obtained by taking the target image features as the material to enhance the reference image features.

[0019] The enhanced target image features are obtained by taking the reference image features as the material to enhance the target image features.

[0020] Specifically, the interleaved perception module includes two attention blocks, each of which is provided with a cross-attention layer, a self-attention layer and a feedforward network in sequence along the image feature input direction.

[0021] Specifically, the expression of the interleaved attention operation is as follows:

[0022] F A′ ←FFN(SelfAttn(CrossAttn(F A ,F B )))

[0023] F B′ ←FFN(SelfAttn(CrossAttn(F B ,F A )))

[0024] Wherein, F A represents the reference image features, and F Bdenote target image features, F A′ denote enhanced reference image features, F B′ denote enhanced target image features.

[0025] Specifically, the expression of the semantic similarity matrix is as follows:

[0026]

[0027] wherein F A′ denote enhanced reference image features, F B′ denote enhanced target image features.

[0028] Specifically, the calculation process of the initial matching flow is as follows:

[0029] A two-dimensional Gaussian kernel is applied to the semantic similarity matrix to suppress non-maximal local maximum, and the semantic similarity matrix is standardized along the last two dimensions to obtain the corresponding correlation distribution;

[0030] The last two dimensions of the semantic similarity matrix are taken as weights, and weighted summation is performed in a two-dimensional coordinate grid to obtain the dense corresponding key point prediction;

[0031] The initial matching flow is calculated by the following formula:

[0032]

[0033]

[0034] wherein P denotes the correlation distribution, G denotes the dense corresponding key point prediction, and M denotes the initial matching flow.

[0035] Specifically, the expression of the local texture similarity matrix is as follows:

[0036]

[0037] wherein p=(x,y) denotes the coordinates of the target pixel in the reference image feature, Δp∈M is the initial predicted matching flow of the target pixel, denote a local neighborhood of a pixel, and r is the neighborhood radius.

[0038] Specifically, in the training process, a joint loss function composed of an endpoint error loss and an auxiliary triplet loss is used to train the matching network to back-propagate and optimize the weight parameters of the matching network, and the expression is as follows:

[0039]

[0040] wherein, denote the auxiliary triplet loss, represents an end-point error loss, and λ1 and λ2 represent weights.

[0041] Specifically, the auxiliary triplet loss takes the features of the positions of the key points in the image as samples for constructing triplets, that is, p={p1, p2,..., p N} and q={q1, q2,..., q N} are respectively the coordinates of the key points in the reference image and the coordinates of the key points in the target image, so as to construct a corresponding loss calculation formula:

[0042]

[0043] wherein, represents the feature of the position of the i-th key point in the reference image, represents the feature of the position of the i-th key point in the target image, represents the feature of the position of a non-corresponding key point in the target image, and N represents the total number of samples.

[0044] Specifically, the expression of the end-point error loss is as follows:

[0045]

[0046] wherein, and are respectively the real matching flow and the predicted matching flow, and N represents the total number of samples.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] The use of the interleaved perception module effectively strengthens the joint modeling of the visual base model on two images, enhances the visual expression capability thereof, and further obtains reliable matching results.

[0049] The matching results are further refined by using an auxiliary loss function and a high-resolution similarity matrix guided upsampling module, wherein the auxiliary loss function is mainly used to distinguish objects with similar semantics, and the high-resolution similarity matrix guided upsampling module uses fine-grained information to make the positioning more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A flowchart of the semantic matching method based on a large-scale pre-training model visual feature provided by the present embodiment;

[0051] Figure 2 A flowchart of the semantic matching network provided by the present embodiment;

[0052] Figure 3 A flowchart of the interleaved perception module provided by the present embodiment. DETAILED DESCRIPTION

[0053] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings of the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0054] As shown in Figure 1 , a semantic matching method provided by the embodiment is provided, which optimizes the visual backbone network DINOv2 by designing a matching module to maximize the potential of the network in the semantic matching task, so as to obtain more accurate matching results and better serve the downstream visual tasks.

[0055] The specific steps are as follows:

[0056] An image data set is obtained, which includes a target image and a reference image corresponding to the semantics.

[0057] A corresponding semantic matching network is constructed based on a visual basic model framework, which includes a feature extraction module, an interleaved perception module, a matching module and an up-sampling module, as shown in Figure 2 The flowchart of the semantic matching network provided by the embodiment is shown.

[0058] The feature extraction module includes a normal image encoder and a high-resolution image encoder, the normal image encoder is used for feature extraction of the input target image and reference image to obtain corresponding target image features and reference image features, and the high-resolution image encoder is used for high-resolution feature extraction of the input target image and reference image to obtain corresponding high-resolution reference image features and high-resolution target image features.

[0059] The interleaved perception module is used for image enhancement of the target image features and the reference image features to output corresponding enhanced reference image features and enhanced target image features.

[0060] The matching module is configured to calculate semantic similarity between the enhanced reference image features and the enhanced target image features to construct a corresponding semantic similarity matrix, and calculate similarity of features between the high-resolution reference image features and the high-resolution target image features to construct a corresponding high-resolution texture similarity matrix;

[0061] The upsampling module is configured to crop the constructed high-resolution texture similarity matrix using the initial matching flow to output a local texture similarity matrix, splice and upsample the local texture similarity matrix and the initial matching flow, and perform a convolution regression operation on the upsampled result to output the predicted matching result of the target image.

[0062] The feature extraction module includes a normal image encoder and a high-resolution image encoder, and each encoder works as follows: given a reference image I A annotated with reference key points K and a target image I The network is designed to predict target key point positions K B corresponding to the reference key points K A for the target image I B . K A may be dense or sparse, depending on the task requirements. The visual backbone network first extracts feature maps F A and F B from the two input images I and I C represents the output dimension of the visual backbone; S represents the step length of the feature map. In this embodiment, all feature maps F A and F B are upsampled to a step length S = 8.

[0063] Figure 3 As shown in the figure, the interleaved perception module provided in this embodiment can significantly enhance the expression ability of features through only a single layer. The input of the interleaved perception module is the feature maps F A and F B . These feature maps pass through two attention blocks in turn, and each block is composed of a cross-attention layer, a self-attention layer and a feedforward network in order. The cross-attention layer is used to fuse features across frames, while the self-attention layer is used to update features within each frame. Unlike previous methods, we find that alternating updates of F A and F B can improve performance. Specifically, in the first attention block of the interleaved attention module, F A serves as the query, while F B serves as the key and value, and F BThe features of F are initially integrated into A′ In contrast, in the subsequent second attention block, F B As a query, and F A As key and value, make F A The features of F are fused into B′ The process can be expressed by the following formula:

[0064] F A′ ←FFN(SelfAttn(CrossAttn(F A ,F B )))

[0065] F B′ ←FFN(SelfAttn(CrossAttn(F B ,F A )))

[0066] Among them, F A represents the reference image feature, F B represents the target image features, F A′ represents the enhanced reference image features, F B′ Represents the enhanced target image features.

[0067] After the interleaved perception module completes feature enhancement, the matching module calculates the semantic similarity matrix based on cosine similarity for the enhanced features to construct the corresponding semantic similarity matrix.

[0068] The specific calculation method is the semantic similarity matrix in As shown below:

[0069]

[0070] C contains the enhanced reference image features F A All pixels in the image are compared with the enhanced target image feature F B The correlation score between all pixels in .

[0071] The following soft-argmax procedure is used to extract prediction keypoints from the correlation matrix C. First, a two-dimensional Gaussian kernel is applied to C to suppress non-maximal local maxima. Then, C is normalized along the last two dimensions to obtain the correlation distribution

[0072] P = Softmax(C).

[0073] Then, using the last two dimensions of C as weights, we can create a grid of Using weighted summation, we can get dense corresponding key point predictions

[0074]

[0075] It is worth noting that the temperature coefficient is introduced to prevent the over-smoothing of the correlation matrix. In this model, the temperature is configured as a learnable parameter, which can be optimized during the training process. Then, the matching flow map can be obtained by the following method

[0076]

[0077] After upsampling the matching flow map to the resolution of the original image, dense correspondences are obtained from each pixel in the source image to the target image. The key points K of sparse prediction are extracted in the middle B Then the key points K can be extracted from P according to the index.

[0078] In this embodiment, a high-resolution texture similarity matrix is used to guide the upsampling operation. In the specific process, a specific local similarity matrix is used instead of a global similarity matrix. This choice is because long-range similarity is unnecessary for upsampling and may have a negative impact on the result, because the high-resolution texture similarity matrix is generated by a shallow network and only contains local information. Local similarity contains the relationship information between the source pixel and the surrounding possible corresponding pixels. Then, the matching flow map and the local similarity are upsampled to the full resolution by bilinear interpolation, and are encoded by the respective convolutional neural networks. Finally, they are connected and input into another convolutional neural network for encoding to obtain the final full-resolution predicted matching flow.

[0079] The semantic matching network is trained using an image dataset to obtain a semantic matching model for image semantic matching.

[0080] In this embodiment, a joint loss function composed of an endpoint error loss and an auxiliary triple loss is used for training. The endpoint error loss is defined as follows:

[0081]

[0082] Where and are the real matching flow and the predicted matching flow, respectively.

[0083] The auxiliary triple loss is defined as follows. The general triple loss is defined as

[0084] L(a,p,n)=max(d(a,p)-d(a,n)+m,0),

[0085] Where a, p and n are the anchor point, positive sample and negative sample of the triplet respectively, d is the distance function (such as L2 distance), and m is the margin. In this invention, we use the features of the key point location as the sample to construct the triplet. Let p = {p1, p2, ..., p N} and q={q1,q2,...,q N} are the coordinates of the source image key points and the target image key points respectively, then the auxiliary function can be defined as

[0086]

[0087] in and They are the features of the location of a key point in the source image, the features of the location of the corresponding key point in the target image, and the features of the location of the non-corresponding key point in the target image. During training, we use weights λ1 and λ2 to balance the two losses, defined as

[0088]

[0089] After obtaining the loss, the model parameters can be optimized through back propagation.

[0090] The target image to be semantically matched is input into the semantic matching model to output a reference image that semantically matches the specified image database to better serve downstream visual tasks.

[0091] In addition, the terms "upper", "lower", "inner", "outer", "front", and "back" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the present invention.

[0092] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of implementation of the present invention. Any equivalent changes or modifications made based on the structure, features and principles described in the scope of the patent application of the present invention should be included in the scope of the patent application of the present invention.

[0093] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the present application, and the protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still make modifications or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features therein, within the technical range disclosed by the present application, and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and all should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A semantic matching method based on visual features of a large-scale pre-trained model, characterized in that: The following steps are involved: Obtain an image dataset, which includes a target image and a reference image corresponding to the semantics; Building a corresponding semantic matching network based on the visual basic model framework, the semantic matching network includes a feature extraction module, an interlaced perception module, a matching module and an upsampling module; The feature extraction module includes a common image encoder and a high-resolution image encoder, wherein the common image encoder is used to perform feature extraction on the input target image and the reference image to obtain corresponding target image features and reference image features, and the high-resolution image encoder is used to perform high-resolution feature extraction on the input target image and the reference image to obtain corresponding high-resolution reference image features and high-resolution target image features; The interlaced perception module is used to perform image enhancement on the target image features and the reference image features to output corresponding enhanced reference image features and enhanced target image features; The interleaved perception module includes two attention blocks, each of which is provided with a cross attention layer, a self-attention layer and a feedforward network in sequence along the image feature input direction; The matching module is used to calculate the semantic similarity between the enhanced reference image features and the enhanced target image features to construct a corresponding semantic similarity matrix, calculate the corresponding initial matching flow based on the semantic similarity matrix, and calculate the feature similarity between the high-resolution reference image features and the high-resolution target image features to construct a corresponding high-resolution texture similarity matrix; The upsampling module uses the initial matching stream to crop the constructed high-resolution texture similarity matrix to output a local texture similarity matrix, concatenates and upsamples the local texture similarity matrix with the initial matching stream, and performs a convolution regression operation on the upsampling result to output a predicted matching result of the target image; The semantic matching network is trained using the image dataset to obtain a semantic matching model for image semantic matching; The reference image, the points to be matched on the reference image, and the target image are input into the semantic matching model to output the matching points on the target image.

2. The semantic matching method based on large-scale pre-trained model visual features according to claim 1 is characterized in that: The interlaced perception module uses an interlaced attention operation to cross-enhance the target image features and the reference image features. The specific operation is as follows: the enhanced reference image features are obtained by enhancing the reference image features using the target image features as the material; The enhanced target image features are obtained by enhancing the target image features using the reference image features as materials.

3. The semantic matching method based on large-scale pre-trained model visual features according to claim 2 is characterized in that: The expression of the staggered attention operation is as follows: ;in, represents the reference image features, represents the target image features, represents the enhanced reference image features, Represents the enhanced target image features.

4. The semantic matching method based on large-scale pre-trained model visual features according to claim 1 is characterized in that: The expression of the semantic similarity matrix is ​​as follows: ;in, represents the enhanced reference image features, Represents the enhanced target image features.

5. The semantic matching method based on large-scale pre-trained model visual features according to claim 1 is characterized in that: The calculation process of the initial matching flow is as follows: A two-dimensional Gaussian kernel is applied to the semantic similarity matrix to suppress non-maximal local maxima, and the semantic similarity matrix is ​​normalized along the last two dimensions to obtain the corresponding correlation distribution; The last two dimensions of the semantic similarity matrix are used as weights and weighted summed in a two-dimensional coordinate grid to obtain densely corresponding key point predictions; The initial matching flow is obtained by calculating using the following formula: ;in, represents the relevant distribution, represents the key point prediction of dense correspondence, Indicates the initial matching flow.

6. The semantic matching method based on large-scale pre-trained model visual features according to claim 5 is characterized in that: The expression of the local texture similarity matrix is ​​as follows: ;in, represents the coordinates of the target pixel in the reference image feature, is the initial predicted matching flow for the target pixel, represents the local neighborhood of a pixel, is the neighborhood radius.

7. The semantic matching method based on large-scale pre-trained model visual features according to claim 1, characterized in that: During the training process, the matching network is trained using a joint loss function consisting of endpoint error loss and auxiliary triplet loss to backpropagate and optimize the weight parameters of the matching network, which is expressed as follows: ;in, represents the auxiliary triplet loss, represents the endpoint error loss, and Represents weight.

8. The semantic matching method based on large-scale pre-trained model visual features according to claim 7, characterized in that: The auxiliary triplet loss takes the features of the key points in the image as samples for constructing triplet, i.e. and are the coordinates of the key points in the reference image and the coordinates of the key points in the target image, respectively, to construct the corresponding loss calculation formula: ;in, Indicates the reference image The features of the key points, Indicates the target image The features of the key points, The features representing the locations of non-corresponding key points in the target image, Indicates the total number of samples.

9. The semantic matching method based on large-scale pre-trained model visual features according to claim 7, characterized in that: The expression of the endpoint error loss is as follows: ;in, and are the real matching flow and the predicted matching flow respectively, Indicates the total number of samples.

Images