Image similarity calculation method and device based on cross-modal attention distillation

Abstract

The application relates to the technical field of artificial intelligence, and provides an image similarity calculation method and device based on cross-modal attention distillation, which comprises the following steps: converting a training image and extracting an encoding vector, fusing multi-modal attention information corresponding to embedding vectors and multi-modal features through an attention alignment network to obtain attention fusion information, iteratively optimizing a visual feature extraction model according to the attention fusion information, extracting vectors of an initial image and a target comparison image respectively after obtaining a target visual feature extraction model based on attention distillation optimization, and calculating image similarity. The application can be applied to an image processing system in the business fields of financial technology, medical health, and endowment, and when image similarity is analyzed, the structural difference between different modal attentions is solved through cross-modal attention alignment, the model is iteratively optimized through attention distillation optimization, and the model training efficiency and the accuracy of similarity analysis are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image similarity calculation method and apparatus based on cross-modal attention distillation. Background Technology

[0002] In recent years, visual Transformer models (such as ViT) have demonstrated superior performance in image recognition tasks through self-attention mechanisms. For example, they acquire image features and determine image similarity based on the similarity of these features. However, the feature learning capabilities of existing visual Transformer models are still limited by the data distribution of a single visual modality. The limitations of existing methods are mainly reflected in the following two aspects: First, existing visual Transformer models struggle to effectively absorb discriminative patterns from non-visual modalities. Second, they cannot perform semantic alignment of multimodal features, making it difficult to achieve good model training results. Third, existing visual models cannot achieve cross-modal feature fusion training, leading to poor model efficiency and training effects. Therefore, existing visual models suffer from poor training efficiency, resulting in inaccurate image similarity analysis. Summary of the Invention

[0003] This invention provides an image similarity calculation method and apparatus based on cross-modal attention distillation, aiming to solve the problem of inaccurate image similarity analysis caused by the poor training efficiency of existing visual models.

[0004] In a first aspect, embodiments of the present invention provide an image similarity calculation method based on cross-modal attention distillation, the method comprising:

[0005] The system receives the input training image and its corresponding multimodal features, and transforms the training image using preset transformation rules to obtain the corresponding embedding vector.

[0006] Extract the encoding vector corresponding to the embedding vector according to the preset visual feature extraction model;

[0007] Multimodal attention information corresponding to the encoded vector and the multimodal features is obtained through a pre-set attention alignment network;

[0008] According to the preset task dynamic selection strategy, multiple sets of attention parameters in the multimodal attention information are fused to obtain the corresponding attention fusion information;

[0009] The visual feature extraction model is iteratively optimized based on a preset gradient descent strategy, the attention fusion information, the encoding vector, and a preset sample set to obtain a target visual feature extraction model optimized by attention distillation.

[0010] According to the conversion rule and the target visual feature extraction model, extract the first contrast vector and the second contrast vector corresponding to the initial image and the target contrast image;

[0011] The image similarity to the first comparison vector and the second comparison vector is calculated according to the preset similarity calculation rules.

[0012] In a second aspect, embodiments of the present invention also provide an image similarity calculation apparatus based on cross-modal attention distillation, the apparatus being used to perform the image similarity calculation method based on cross-modal attention distillation as described in the first aspect above, the apparatus comprising:

[0013] The embedding vector acquisition unit is used to receive the input training image and the corresponding multimodal features, and to convert the training image according to the preset conversion rules to obtain the corresponding embedding vector.

[0014] The encoding vector extraction unit is used to extract the encoding vector corresponding to the embedding vector according to a preset visual feature extraction model;

[0015] A multimodal attention information acquisition unit is used to acquire multimodal attention information corresponding to the encoding vector and the multimodal features through a preset attention alignment network;

[0016] The attention fusion information acquisition unit is used to fuse multiple sets of attention parameters in the multimodal attention information according to a preset task dynamic selection strategy to obtain the corresponding attention fusion information.

[0017] An iterative optimization unit is used to iteratively optimize the visual feature extraction model according to a preset gradient descent strategy, the attention fusion information, the encoding vector and a preset sample set, so as to obtain a target visual feature extraction model based on attention distillation optimization.

[0018] The contrast vector acquisition unit is used to extract a first contrast vector and a second contrast vector corresponding to the initial image and the target contrast image according to the conversion rule and the target visual feature extraction model.

[0019] The image similarity calculation unit is used to calculate the image similarity with the first comparison vector and the second comparison vector according to the preset similarity calculation rules.

[0020] Thirdly, embodiments of the present invention also provide an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect above.

[0021] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the method described in the first aspect.

[0022] This invention provides an image similarity calculation method, apparatus, device, and medium based on cross-modal attention distillation. The method includes: transforming a training image and extracting an encoded vector; using an attention alignment network to embed the vector and multimodal attention information corresponding to multimodal features; fusing multiple sets of attention parameters in the multimodal attention information to obtain attention fusion information; iteratively optimizing a visual feature extraction model based on the attention fusion information to obtain a target visual feature extraction model optimized by attention distillation; and extracting a first contrast vector and a second contrast vector from the initial image and the target contrast image respectively using the target visual feature extraction model and calculating image similarity. This invention addresses the structural differences between different modal attention through cross-modal attention alignment and iteratively optimizes the visual feature extraction model through attention distillation to improve model training efficiency and enhance the semantic perception capability of image similarity calculation, thereby improving the accuracy and reliability of image similarity calculation. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating the image similarity calculation method based on cross-modal attention distillation provided in this embodiment of the invention;

[0025] Figure 2 A schematic block diagram of an image similarity calculation device based on cross-modal attention distillation provided in an embodiment of the present invention;

[0026] Figure 3 A schematic block diagram of an electronic device provided in an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram illustrating an application scenario of the image similarity calculation method based on cross-modal attention distillation provided in this embodiment of the invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0030] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0031] It should also be further understood that the term "and / or" as used in this specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. Embodiments of this invention provide a method and apparatus for calculating image similarity based on cross-modal attention distillation. For application scenarios of this image similarity calculation method based on cross-modal attention distillation, please refer to... Figure 4 , Figure 4 This is a schematic diagram illustrating an application scenario of the image similarity calculation method based on cross-modal attention distillation provided in this invention. The image similarity calculation method based on cross-modal attention distillation is applied in scenarios such as... Figure 4In this application scenario, the computer device communicates with at least one host device via a network. The computer device can be a server or a personal computer. The server can be a standalone server or a server cluster composed of multiple servers. The host device can be, but is not limited to, electronic devices such as servers, smartphones, tablets, and desktop computers. The computer device executes an image similarity calculation method based on cross-modal attention distillation and interacts with the host device. The host device can send training images and images to be analyzed to the computer device. The computer device performs iterative optimization of the model and obtains the image similarity through corresponding analysis, and then feeds back the image similarity to the host device. In addition to being applied to the smart city field for similarity analysis of road images, the image similarity calculation method based on cross-modal attention distillation in this application embodiment can also be applied to the financial insurance and medical fields. For example, in the medical field, an image target visual feature extraction model corresponding to the above-mentioned image similarity calculation method based on cross-modal attention distillation can be constructed to perform similarity analysis on medical monitoring images obtained from medical testing; similarly, in the financial insurance field, a target visual feature extraction model corresponding to the above-mentioned image similarity calculation method based on cross-modal attention distillation can be constructed to perform similarity analysis on images containing human figures. The invention will now be described in detail through specific embodiments.

[0032] Figure 1 This is a flowchart illustrating the image similarity calculation method based on cross-modal attention distillation provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110-S170.

[0033] S110. Receive the input training image and corresponding multimodal features, and convert the training image according to a preset conversion rule to obtain the corresponding embedding vector.

[0034] The system can input a training image and its corresponding multimodal features. These multimodal features can be text features corresponding to the training image. For example, the text accompanying an image displayed on social media can be used as the input training image; extracting the text features yields the corresponding text features. Similarly, the audio content of a video displayed on social media can be converted into text and its features extracted to obtain the corresponding text features; the video's cover image can then be used as the input training image. Multimodal features are non-image features closely related to the training image. "Modality" refers to the category of the acquired feature information; for example, the encoded vector obtained from the training image represents one modality, while the text features represent another.

[0035] First, the training image is transformed using transformation rules to obtain the embedding vector corresponding to the training image.

[0036] In one embodiment, step S110 includes: performing image segmentation on the training image according to the segmentation parameters in the transformation rule to obtain corresponding image segmentation information; and mapping the image segmentation information to the corresponding multidimensional embedding space according to the linear projection matrix in the transformation rule to obtain the corresponding embedding vector.

[0037] Specifically, the training image can be segmented using the segmentation parameters in the transformation rules. These parameters can set the size of the segmentation region. For example, if the segmentation parameter is set to P×P, the training image can be segmented according to this parameter to obtain multiple segmented images corresponding to the size of P×P. If the input training image is I, it can be segmented into N non-overlapping segmented images according to the segmentation parameters. The size of each segmented image is P×P, and the resulting multiple segmented images constitute the image segmentation information.

[0038] Furthermore, the image segmentation information is mapped to the corresponding multidimensional embedding space based on the linear projection matrix in the transformation rule. Specifically, the linear projection matrix can be represented as follows: Where E is the linear projection matrix, C is the number of channels, and P 2 That is, corresponding to the above segmentation parameters P×P, where r is the dimension of the embedding space. The image segmentation information is then mapped to an r-dimensional embedding space, and the resulting embedding vector is expressed by formula (1):

[0039]

[0040] Where, p i x represents the pixel value of the i-th segmented image. i Let E be the mapping vector corresponding to the i-th segmented image block, and let E be the linear projection matrix. If the position encoding vector is given, then through the above mapping process, a set of vector sequences X = [x1, x2, ..., x] corresponding to the training image can be obtained. N This vector sequence can also serve as a set of embedding vectors corresponding to the training vectors.

[0041] S120. Extract the encoding vector corresponding to the embedding vector according to the preset visual feature extraction model.

[0042] The embedding vectors obtained from the above steps are input into the visual feature extraction model. The model then uses its configured Transformer encoder to extract the corresponding encoding vectors, which can be used to represent the image features of the training image. The visual feature extraction model also includes an attention network that operates on the Transformer encoder. During the encoding vector extraction process, the attention network helps the Transformer encoder focus more on key local regions within the embedding vectors, resulting in encoding vectors that better reflect the features of the training image. The attention network contains multiple attention nodes, each corresponding to an attention parameter value. Specifically, this attention network can be a self-attention network.

[0043] S130. Obtain multimodal attention information corresponding to the encoded vector and the multimodal features through a preset attention alignment network.

[0044] Furthermore, multimodal attention information corresponding to the embedding vector and multimodal features is obtained through an attention alignment network. This step requires aligning the visual attention corresponding to the embedding vector with the attention distribution corresponding to the multimodal features, that is, semantically aligning the visual attention with the multimodal features to obtain multimodal attention information.

[0045] In one embodiment, step S130 includes: obtaining the visual attention corresponding to the feature extraction process of the encoded vector; mapping the attention distribution of the multimodal features to the attention space corresponding to the visual attention to obtain the corresponding multimodal mapped attention; and combining the visual attention and the multimodal mapped attention to obtain the corresponding multimodal attention information.

[0046] Specifically, the attention parameter values ​​used in the feature extraction process of the encoding vector can be obtained from the attention network of the visual feature extraction model, and the attention parameter values ​​corresponding to the encoding vector can be used as visual attention. For example, visual attention can be expressed by formula (2):

[0047]

[0048] Where Q, K, and V are the lookup, key, and value matrices, respectively, T is the matrix transpose operation, softmax is the normalization function in formula (2), and d k This represents the dimensions of the query vector Q and the key vector K.

[0049] During the extraction of multimodal features, the corresponding attention parameter values ​​can also be obtained. These attention parameter values ​​can then be used as the attention distribution of the multimodal features. Further, the attention distribution of the multimodal features is mapped to the attention space corresponding to visual attention, thus obtaining the multimodal mapped attention corresponding to the attention distribution of the multimodal features. First, the bias term parameters of the attention space corresponding to visual attention can be determined. Based on the projection matrix and the bias term parameters, the attention distribution is mapped to the attention space corresponding to visual attention. The specific mapping process can be represented by formula (3):

[0050]

[0051] Wherein, GELU (Gaussian Error Linear Unit) is the activation function, and its mathematical expression is: GELU(x)=x·Φ(x), where Φ(x) represents the cumulative distribution function of the standard normal distribution; It is a learnable projection matrix (trainable through iterative optimization), where b1 and b2 are bias term parameters. This represents the attention distribution corresponding to the multimodal features. Through the above mapping process, the attention distribution of the multimodal features (i.e., the teacher's attention distribution) can be mapped to a compatible space corresponding to visual attention, thus obtaining the corresponding multimodal mapped attention.

[0052] The visual attention and multimodal mapping attention obtained from the above steps are combined to form the corresponding multimodal attention information, which contains two sets of attention parameters.

[0053] S140. Based on a preset task dynamic selection strategy, multiple sets of attention parameters in the multimodal attention information are fused to obtain corresponding attention fusion information.

[0054] Based on the task dynamic selection strategy, multiple sets of attention parameters in the multimodal attention information are fused, that is, two sets of attention parameters in the multimodal attention information are fused to obtain attention fusion information.

[0055] In one embodiment, step S140 includes: calculating a weight factor corresponding to the initial task loss value and the multimodal attention information according to the weight factor calculation formula in the task dynamic selection strategy; and fusing multiple sets of attention parameters in the multimodal attention information according to the weight factor to obtain corresponding attention fusion information.

[0056] The task dynamic selection strategy is configured with a weight factor calculation formula. The weight factor corresponding to the initial task loss value and multimodal attention information can be calculated according to the weight factor calculation formula. The weight factor calculation formula can be expressed by formula (4):

[0057]

[0058] Where λ is the calculated weighting factor, and σ is the sigmoid function. This represents the initial task loss value corresponding to the task loss function, N is the number of segmented images obtained from the training image segmentation, and i and j are array indices. The multimodal attention information includes visual attention A and multimodal mapping attention. (That is, the formula (3) above is used to calculate) Both can be represented using two-dimensional arrays, and since the two two-dimensional arrays are of equal size after the alignment process described above, we can uniquely determine the corresponding attention parameter value in the two-dimensional array by using i as the horizontal coordinate parameter and j as the vertical coordinate parameter.

[0059] Furthermore, multiple sets of attention parameters in the multimodal attention information are fused according to the weighting factors. The specific fusion can be expressed by formula (5):

[0060]

[0061] Where A represents the attention fusion information obtained through fusion, and λ is the weighting factor. For visual attention, For multimodal mapping attention.

[0062] S150. The visual feature extraction model is iteratively optimized according to the preset gradient descent strategy, the attention fusion information, the encoding vector and the preset sample set to obtain the target visual feature extraction model based on attention distillation optimization.

[0063] The visual feature extraction model is iteratively optimized based on gradient descent strategy, attention fusion information, encoding vector and pre-set sample set. During the iterative optimization process, the attention network in the visual feature extraction model needs to be distilled to obtain the target visual feature extraction model based on attention distillation optimization.

[0064] In one embodiment, step S150 includes: obtaining similarity values ​​between a set of sample training features in the sample set and the encoding vector; calculating a task loss value corresponding to the similarity value according to the contrastive loss function in the gradient descent strategy; updating the weight factor corresponding to the attention fusion information according to the weight factor calculation formula and the task loss value; dynamically adjusting the attention fusion information according to the updated weight factor to obtain corresponding attention optimization information; configuring the parameters of the attention network in the visual feature extraction model according to the attention optimization information; extracting the encoding vector corresponding to the embedding vector again according to the reconfigured visual feature extraction model, and returning to the step of obtaining similarity values ​​between a set of sample training features in the sample set and the encoding vector for iterative optimization until a preset training termination condition is met, then terminating the iterative optimization and outputting the current visual feature extraction model as the target visual feature extraction model.

[0065] Specifically, a set of sample training features can be obtained from the sample set, and the similarity value between the set of sample training features and the corresponding encoding vector can be calculated. The sample set contains multiple sets of sample training features. Each set of sample training features includes one positive sample feature. and multiple negative sample features Positive sample features are the features corresponding to positive sample images that are highly similar to the training images. Positive sample features are obtained by combining the visual and non-visual features of the positive sample images (such as the text features corresponding to the positive sample images). Negative sample features are the features corresponding to negative sample images that are dissimilar to the training images. Negative sample features are obtained by combining the visual and non-visual features of the dissimilar negative sample images (such as the text features corresponding to the negative sample images). Since the length of the encoding vector is equal to the lengths of the positive and negative sample features, the similarity values ​​between the encoding vector and the positive sample features, and between the encoding vector and the negative sample features, can be calculated using the cosine similarity formula.

[0066] The task loss value corresponding to the similarity value is calculated based on the ratio loss function. The specific process of obtaining the task loss value can be expressed by formula (6):

[0067]

[0068] Where exp is the exponentiation based on the natural constant e, and z I This is the encoding vector corresponding to image I. This represents the similarity value between the encoded vector and the features of the positive samples. Let be the similarity value between the encoded vector and the negative sample features, M be the total number of negative sample features, and τ be the temperature parameter. This is the calculated task loss value.

[0069] The weight factors corresponding to the attention fusion information are updated according to the weight factor calculation formula and the task loss value. That is, based on the above weight factor calculation formula, the initial task loss value is replaced by the task loss value, and the weight factor corresponding to the task loss value is recalculated to realize the weight factor update. The two sets of attention parameters in the attention fusion information are dynamically adjusted according to the updated weight factors. Based on the above formula (5), the two sets of attention parameters are re-fused with the new weight factors to obtain the corresponding attention optimization information.

[0070] The attention optimization information contains the optimized parameter values ​​corresponding to the attention network. This information allows for parameter configuration of the attention network, specifically adjusting the attention parameter values ​​of each attention node. Based on the reconfigured visual feature extraction model, the encoding vector corresponding to the embedding vector is extracted again. If the training termination condition is not met, the process returns to obtaining the similarity values ​​between a set of training features from the sample set and the encoding vector. This iterative optimization process effectively distills the attention network within the visual feature extraction model.

[0071] If the training termination condition is met, the iterative optimization process can be terminated, and the currently trained visual feature extraction model can be output as the target visual feature extraction model. The training termination condition can be a training iteration threshold; when the number of iterations reaches this threshold, the training termination condition is met, and the iterative optimization process terminates. Alternatively, the training termination condition can be a loss value threshold; when the calculated task loss value is less than this threshold, the training termination condition is met, and the iterative optimization process terminates.

[0072] In one embodiment, before updating the weight factor corresponding to the attention fusion information according to the weight factor calculation formula and the task loss value, the method further includes: determining the temperature parameter corresponding to the task loss value according to a preset temperature parameter adjustment strategy; and configuring the weight factor calculation formula according to the determined temperature parameter.

[0073] In the aforementioned iterative optimization process, the temperature parameter can be preset to a fixed value, such as setting the temperature parameter τ to 0.08. While setting the temperature parameter to a fixed value can meet basic model training requirements, it is not conducive to efficient model training. Therefore, to accelerate model convergence and improve training efficiency, the temperature parameter can be dynamically adjusted before updating the weight factors. For example, a temperature parameter adjustment strategy can be used to determine the temperature parameter corresponding to the task loss value, and the determined temperature parameter can be used to configure the weight factor calculation formula, thereby achieving dynamic adjustment of the temperature parameter for different training stages and different training effects.

[0074] In one embodiment, determining the temperature parameter corresponding to the task loss value according to a preset temperature parameter adjustment strategy includes: calculating the recorded historical loss value according to the loss value fluctuation function in the temperature parameter adjustment strategy to obtain the corresponding loss value fluctuation coefficient; the historical loss value is the task loss value recorded in each iteration optimization process; and obtaining the matching temperature parameter from the temperature parameter adjustment strategy according to the iteration optimization number and the loss value fluctuation coefficient.

[0075] The loss value fluctuation coefficient can be calculated based on the recorded historical loss values. The historical loss values ​​are the task loss values ​​obtained from each iteration of optimization, and thus contain multiple task loss values. The loss value fluctuation function can be expressed by formula (7):

[0076]

[0077] T represents the number of the latest task loss values ​​used to calculate the fluctuation coefficient of the loss value, b i Let b be the loss value of the i-th task, and b0 be the average of the latest T task loss values. P is the calculated loss value fluctuation coefficient.

[0078] Based on the iteration optimization number and the loss value fluctuation coefficient, the corresponding temperature parameters can be obtained from the temperature parameter adjustment strategy. The iteration optimization number is the number of times the visual feature extraction model is iterated and optimized. Specifically, the temperature parameter adjustment strategy includes multiple parameter intervals, each corresponding to an iteration number interval. For example, if the iteration number interval is [1, 20], the corresponding parameter interval is [0.8, 0.51]; if the iteration number interval is [21, 100], the corresponding parameter interval is [0.5, 0.15]; and if the iteration number interval is [101, 1000], the corresponding parameter interval is [0.12, 0.06]. A parameter interval can be obtained by matching the current iteration optimization number, and the initial parameters corresponding to the iteration optimization number can be further determined within the matched parameter interval. For example, the initial parameters corresponding to the iteration optimization number [0.5, 0.15] can be expressed by formula (8):

[0079]

[0080] Where f is the initial parameter and v is the number of iterations. Further, it is determined whether the fluctuation coefficient of the loss value is larger than the previously calculated fluctuation coefficient. If the current fluctuation coefficient is larger than the previously calculated fluctuation coefficient, the initial parameter f is increased accordingly to obtain the final configurable temperature parameter. Alternatively, the ratio between the current and previously calculated fluctuation coefficients is calculated, and this ratio is multiplied by the increased initial parameter f, which is then used as the temperature parameter. If the current fluctuation coefficient is not larger than the previously calculated fluctuation coefficient, the obtained initial parameter is directly used as the configurable temperature parameter.

[0081] S160. Extract the first contrast vector and the second contrast vector corresponding to the initial image and the target contrast image according to the conversion rule and the target visual feature extraction model.

[0082] After the attention distillation optimization described above, features can be extracted from the input initial image and the target comparison image using the transformation rules and the obtained target visual feature extraction model. First, the initial image and the target comparison image are transformed using the transformation rules respectively. Then, features are extracted from the embedded vectors obtained after transformation based on the target visual feature extraction model, thereby obtaining the first comparison vector and the second comparison vector. The specific process of obtaining the first comparison vector and the second comparison vector is similar to steps S110 and S120, and will not be described in detail here.

[0083] S170. Calculate the image similarity with the first comparison vector and the second comparison vector according to the preset similarity calculation rules.

[0084] Further, the image similarity with the first comparison vector and the second comparison vector is calculated according to the similarity calculation rules. The similarity calculation rules can be the cosine similarity formula or other similarity calculation formulas. Then, the cosine similarity formula can be used to calculate the similarity between the first comparison vector and the second comparison vector, so as to obtain the image similarity that reflects the similarity between the initial image and the target comparison image. The greater the image similarity, the higher the similarity between the initial image and the target comparison image.

[0085] This application discloses an image similarity calculation method based on cross-modal attention distillation. The method includes: transforming a training image and extracting an encoded vector; using an attention alignment network to embed the vector and multimodal attention information corresponding to multimodal features; fusing multiple sets of attention parameters in the multimodal attention information to obtain attention fusion information; iteratively optimizing a visual feature extraction model based on the attention fusion information to obtain a target visual feature extraction model optimized by attention distillation; and extracting a first comparison vector and a second comparison vector from the initial image and the target comparison image respectively using the target visual feature extraction model and calculating the image similarity. This invention addresses the structural differences between different modal attention through cross-modal attention alignment and iteratively optimizes the visual feature extraction model through attention distillation to improve model training efficiency and enhance the semantic perception capability of image similarity calculation, thereby improving the accuracy and reliability of image similarity calculation.

[0086] Figure 2 This is a schematic block diagram of an image similarity calculation device based on cross-modal attention distillation provided in an embodiment of the present invention. Figure 2 As shown, corresponding to the above image similarity calculation method based on cross-modal attention distillation, the present invention also provides an image similarity calculation device based on cross-modal attention distillation, the device being configured in, as shown in, Figure 4 In this application scenario, the computer device communicates with at least one host device via a network. The computer device can be a server or a personal computer. The server can be a standalone server or a server cluster consisting of multiple servers. The host device can be, but is not limited to, electronic devices such as servers, smartphones, tablets, and desktop computers. For details, please refer to [link to relevant documentation]. Figure 2 The image similarity calculation device 700 based on cross-modal attention distillation includes:

[0087] The embedding vector acquisition unit 701 is used to receive the input training image and the corresponding multimodal features, and to convert the training image according to the preset conversion rules to obtain the corresponding embedding vector.

[0088] In some embodiments, when the embedding vector acquisition unit 701 performs the step of converting the training image according to a preset conversion rule to obtain the corresponding embedding vector, it is specifically used to: perform image segmentation on the training image according to the segmentation parameters in the conversion rule to obtain the corresponding image segmentation information; and map the image segmentation information to the corresponding multidimensional embedding space according to the linear projection matrix in the conversion rule to obtain the corresponding embedding vector.

[0089] The encoding vector extraction unit 702 is used to extract the encoding vector corresponding to the embedding vector according to a preset visual feature extraction model.

[0090] The multimodal attention information acquisition unit 703 is used to acquire multimodal attention information corresponding to the encoding vector and the multimodal features through a preset attention alignment network.

[0091] In some embodiments, when the multimodal attention information acquisition unit 703 performs the step of acquiring multimodal attention information corresponding to the encoding vector and the multimodal features through a preset attention alignment network, it is specifically used to: acquire the visual attention corresponding to the feature extraction process of the encoding vector; map the attention distribution of the multimodal features to the attention space corresponding to the visual attention to obtain the corresponding multimodal mapped attention; and combine the visual attention and the multimodal mapped attention to obtain the corresponding multimodal attention information.

[0092] The attention fusion information acquisition unit 704 is used to fuse multiple sets of attention parameters in the multimodal attention information according to a preset task dynamic selection strategy to obtain corresponding attention fusion information.

[0093] In some embodiments, when the attention fusion information acquisition unit 704 performs the step of fusing multiple sets of attention parameters in the multimodal attention information according to a preset task dynamic selection strategy to obtain corresponding attention fusion information, it is specifically used to: calculate a weight factor corresponding to the initial task loss value and the multimodal attention information according to the weight factor calculation formula in the task dynamic selection strategy; and fuse multiple sets of attention parameters in the multimodal attention information according to the weight factor to obtain corresponding attention fusion information.

[0094] The iterative optimization unit 705 is used to iteratively optimize the visual feature extraction model according to the preset gradient descent strategy, the attention fusion information, the encoding vector and the preset sample set, so as to obtain the target visual feature extraction model based on attention distillation optimization.

[0095] In some embodiments, when the iterative optimization unit 705 performs the step of iteratively optimizing the visual feature extraction model according to a preset gradient descent strategy, the attention fusion information, the encoding vector, and a preset sample set to obtain a target visual feature extraction model based on attention distillation optimization, it is specifically configured to: obtain similarity values ​​between a set of sample training features in the sample set and the encoding vector; calculate the task loss value corresponding to the similarity value according to the contrast loss function in the gradient descent strategy; update the weight factor corresponding to the attention fusion information according to the weight factor calculation formula and the task loss value; dynamically adjust the attention fusion information according to the updated weight factor to obtain corresponding attention optimization information; configure the parameters of the attention network in the visual feature extraction model according to the attention optimization information; extract the encoding vector corresponding to the embedding vector again according to the reconfigured visual feature extraction model, and return to perform the step of obtaining the similarity values ​​between a set of sample training features in the sample set and the encoding vector for iterative optimization, until a preset training termination condition is met, then terminate the iterative optimization and output the current visual feature extraction model as the target visual feature extraction model.

[0096] In some embodiments, before performing the step of updating the weight factor corresponding to the attention fusion information according to the weight factor calculation formula and the task loss value, the iterative optimization unit 705 is further configured to perform the following steps: determining the temperature parameter corresponding to the task loss value according to a preset temperature parameter adjustment strategy; and configuring the weight factor calculation formula according to the determined temperature parameter.

[0097] In some embodiments, when the iterative optimization unit 705 performs the step of determining the temperature parameter corresponding to the task loss value according to the preset temperature parameter adjustment strategy, it is specifically used to: calculate the recorded historical loss value according to the loss value fluctuation function in the temperature parameter adjustment strategy to obtain the corresponding loss value fluctuation coefficient; the historical loss value is the task loss value recorded in each iterative optimization process; and obtain the matching temperature parameter from the temperature parameter adjustment strategy according to the number of iterative optimizations and the loss value fluctuation coefficient.

[0098] The contrast vector acquisition unit 706 is used to extract a first contrast vector and a second contrast vector corresponding to the initial image and the target contrast image according to the conversion rule and the target visual feature extraction model.

[0099] The image similarity calculation unit 707 is used to calculate the image similarity with the first comparison vector and the second comparison vector according to the preset similarity calculation rules.

[0100] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned image similarity calculation device and its units based on cross-modal attention distillation can be referred to the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and brevity, these will not be repeated here.

[0101] This application discloses an image similarity calculation device based on cross-modal attention distillation for executing the aforementioned image similarity calculation method based on cross-modal attention distillation. The method includes: transforming a training image and extracting an encoded vector; using an attention alignment network to embed the vector and multimodal attention information corresponding to multimodal features; fusing multiple sets of attention parameters in the multimodal attention information to obtain attention fusion information; iteratively optimizing a visual feature extraction model based on the attention fusion information to obtain a target visual feature extraction model optimized based on attention distillation; and extracting a first comparison vector and a second comparison vector from the initial image and the target comparison image respectively using the target visual feature extraction model and calculating the image similarity. This invention addresses the structural differences between different modal attention through cross-modal attention alignment and iteratively optimizes the visual feature extraction model through attention distillation optimization to improve model training efficiency and enhance the semantic perception capability of image similarity calculation, thereby improving the accuracy and reliability of image similarity calculation.

[0102] The aforementioned image similarity calculation device based on cross-modal attention distillation can be implemented as a computer program, which can be used in, for example... Figure 3 It runs on the electronic device shown.

[0103] Please see Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 800 can be a terminal or a server. The terminal can be an electronic device with communication functions. The server can be a standalone server or a server cluster composed of multiple servers.

[0104] See Figure 3 The electronic device 800 includes a processor 802, a memory, and a network interface 805 connected via a system bus 801. The memory may include a non-volatile storage medium 803 and internal memory 804.

[0105] The non-volatile storage medium 803 may store an operating system 8031 ​​and a computer program 8032. The computer program 8032 includes program instructions that, when executed, cause the processor 802 to perform an image similarity calculation method based on cross-modal attention distillation.

[0106] The processor 802 provides computing and control capabilities to support the operation of the entire electronic device 800.

[0107] The internal memory 804 provides an environment for the execution of the computer program 8032 in the non-volatile storage medium 803. When the computer program 8032 is executed by the processor 802, the processor 802 can execute an image similarity calculation method based on cross-modal attention distillation.

[0108] This network interface 805 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device 800 to which the present invention is applied. The specific electronic device 800 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0109] The processor 802 is used to run the computer program 8032 stored in the memory to implement the steps included in the above-described image similarity calculation method based on cross-modal attention distillation.

[0110] It should be understood that, in this embodiment of the invention, the processor 802 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0111] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0112] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the steps included in the image similarity calculation method based on cross-modal attention distillation described above.

[0113] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0115] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0116] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0118] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image similarity calculation method based on cross-modal attention distillation, characterized in that, The method includes: The system receives the input training image and its corresponding multimodal features, and transforms the training image using preset transformation rules to obtain the corresponding embedding vector. Extract the encoding vector corresponding to the embedding vector according to the preset visual feature extraction model; Multimodal attention information corresponding to the encoded vector and the multimodal features is obtained through a pre-set attention alignment network; According to the preset task dynamic selection strategy, multiple sets of attention parameters in the multimodal attention information are fused to obtain the corresponding attention fusion information; The visual feature extraction model is iteratively optimized based on a preset gradient descent strategy, the attention fusion information, the encoding vector, and a preset sample set to obtain a target visual feature extraction model optimized by attention distillation. According to the conversion rule and the target visual feature extraction model, extract the first contrast vector and the second contrast vector corresponding to the initial image and the target contrast image; The image similarity to the first comparison vector and the second comparison vector is calculated according to the preset similarity calculation rules.

2. The image similarity calculation method based on cross-modal attention distillation according to claim 1, characterized in that, The step of transforming the training image using a preset transformation rule to obtain the corresponding embedding vector includes: The training image is segmented according to the segmentation parameters in the conversion rule to obtain the corresponding image segmentation information; The image segmentation information is mapped to the corresponding multidimensional embedding space according to the linear projection matrix in the transformation rule to obtain the corresponding embedding vector.

3. The image similarity calculation method based on cross-modal attention distillation according to claim 1, characterized in that, The step of obtaining multimodal attention information corresponding to the encoded vector and the multimodal features through a pre-set attention alignment network includes: Obtain the visual attention corresponding to the feature extraction process of the encoded vector; The attention distribution of the multimodal features is mapped to the attention space corresponding to visual attention to obtain the corresponding multimodal mapped attention; The visual attention is combined with the multimodal mapping attention to form the corresponding multimodal attention information.

4. The image similarity calculation method based on cross-modal attention distillation according to any one of claims 1-3, characterized in that, The step of fusing multiple sets of attention parameters in the multimodal attention information according to a preset task dynamic selection strategy to obtain corresponding attention fusion information includes: The weight factor corresponding to the initial task loss value and the multimodal attention information is calculated according to the weight factor calculation formula in the task dynamic selection strategy. The attention parameters in the multimodal attention information are fused according to the weighting factor to obtain the corresponding attention fusion information.

5. The image similarity calculation method based on cross-modal attention distillation according to claim 4, characterized in that, The step of iteratively optimizing the visual feature extraction model based on a preset gradient descent strategy, the attention fusion information, the encoding vector, and a preset sample set to obtain a target visual feature extraction model optimized by attention distillation includes: Obtain the similarity value between a set of training features of samples in the sample set and the encoding vector; The task loss value corresponding to the similarity value is calculated based on the contrastive loss function in the gradient descent strategy. The weight factors corresponding to the attention fusion information are updated according to the weight factor calculation formula and the task loss value; The attention fusion information is dynamically adjusted based on the updated weighting factors to obtain the corresponding attention optimization information. Configure the parameters of the attention network in the visual feature extraction model based on the attention optimization information; The visual feature extraction model with reconfigured parameters is used to extract the encoding vector corresponding to the embedding vector again, and the step of obtaining the similarity value between a set of sample training features in the sample set and the encoding vector is returned to perform iterative optimization until the preset training termination condition is met, then the iterative optimization is terminated and the current visual feature extraction model is output as the target visual feature extraction model.

6. The image similarity calculation method based on cross-modal attention distillation according to claim 5, characterized in that, Before updating the weight factors corresponding to the attention fusion information according to the weight factor calculation formula and the task loss value, the method further includes: The temperature parameter corresponding to the task loss value is determined based on the preset temperature parameter adjustment strategy; The weighting factor calculation formula is configured with parameters based on the determined temperature parameters.

7. The image similarity calculation method based on cross-modal attention distillation according to claim 6, characterized in that, The step of determining the temperature parameter corresponding to the task loss value based on the preset temperature parameter adjustment strategy includes: The historical loss values ​​are calculated based on the loss value fluctuation function in the temperature parameter adjustment strategy to obtain the corresponding loss value fluctuation coefficient; the historical loss value is the task loss value recorded in each iteration of optimization. The temperature parameter that matches the temperature parameter adjustment strategy is obtained based on the number of iterations and the loss value fluctuation coefficient.

8. An image similarity calculation device based on cross-modal attention distillation, characterized in that, The apparatus is used to perform the image similarity calculation method based on cross-modal attention distillation as described in any one of claims 1-7, the apparatus comprising: The embedding vector acquisition unit is used to receive the input training image and the corresponding multimodal features, and to convert the training image according to the preset conversion rules to obtain the corresponding embedding vector. The encoding vector extraction unit is used to extract the encoding vector corresponding to the embedding vector according to a preset visual feature extraction model; A multimodal attention information acquisition unit is used to acquire multimodal attention information corresponding to the encoding vector and the multimodal features through a preset attention alignment network; The attention fusion information acquisition unit is used to fuse multiple sets of attention parameters in the multimodal attention information according to a preset task dynamic selection strategy to obtain the corresponding attention fusion information. An iterative optimization unit is used to iteratively optimize the visual feature extraction model according to a preset gradient descent strategy, the attention fusion information, the encoding vector and a preset sample set, so as to obtain a target visual feature extraction model based on attention distillation optimization. The contrast vector acquisition unit is used to extract a first contrast vector and a second contrast vector corresponding to the initial image and the target contrast image according to the conversion rule and the target visual feature extraction model. The image similarity calculation unit is used to calculate the image similarity with the first comparison vector and the second comparison vector according to the preset similarity calculation rules.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image similarity calculation method based on cross-modal attention distillation as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the image similarity calculation method based on cross-modal attention distillation as described in any one of claims 1-7.

Images