A graph-text matching method in multi-object scenarios based on heterogeneous information interaction

Abstract

A method for image-text matching in a multi-object scenario based on heterogeneous information interaction belongs to the field of image-text matching in multimedia information processing. The present invention first divides the data set into a training set and a test set. Then, a joint hash code matrix learning formula for the multi-object data image-text matching problem is constructed. Then, the joint hash code matrix learning formula is solved using the alternating update algorithm 1 to obtain the joint hash code matrix B. Then, a hash function learning formula for the multi-object data image-text matching problem is constructed. Then, the hash function learning formula is solved using the alternating update algorithm 2 to obtain hash functions F1 and F2. Finally, the test set is tested using the joint hash code matrix B and hash functions F1 and F2 obtained in the above steps and the average accuracy of image-text matching in the multi-object scenario is calculated. Compared with other methods, the average accuracy of the present invention is higher.

Description

Technical Field

[0001] The present invention belongs to the field of image-text matching in multimedia information processing, and specifically relates to an image-text matching method in a multi-object scenario based on heterogeneous information interaction. Background Art

[0002] In the field of multimedia information processing, cross-modal matching refers to matching between different modalities. Typically, one modality is used as a query sample, and an algorithm is used to match data from another modality. Images and text are widely used and easily accessible data modalities, making image-text matching a major research area in cross-modal matching. Due to the limited amount of content on most social media platforms, people often share image and text information through information splicing and fusion. For example, when posting on WeChat Moments, people often do not post multiple images with corresponding text descriptions. Instead, they tend to splice photos together and add one or more rich text descriptions, thereby sharing as much information as possible within the limited content. This results in a single text or image containing not just information about a single item, but rather complex information about multiple items. This type of image-text presentation, which includes multiple items, has become the preferred method for most people when sharing. Due to the richness of this information, the processing and matching of this data is more complex. Therefore, accurately matching images and text in multi-object scenarios is a valuable problem. However, existing image-text matching methods perform poorly in this scenario, and effective novel technical methods are needed to solve the image-text matching problem in multi-object scenarios. Summary of the Invention

[0003] In order to solve the above problems, the present invention provides a method for image-text matching in a multi-object scenario based on heterogeneous information interaction, the method comprising the steps of:

[0004] Divide the dataset into training set and test set.

[0005] Construct a joint hash code matrix learning formula for the multi-object data graph-text matching problem.

[0006] The joint hash code matrix learning formula is solved using the alternating update algorithm 1 to obtain the joint hash code matrix B.

[0007] Construct a hash function learning formula for the multi-object data graph-text matching problem.

[0008] The hash function learning formula is solved using the alternating update algorithm 2 to obtain hash functions F1 and F2.

[0009] The joint hash code matrix B and hash functions F1 and F2 obtained in the above steps are used to test the test set and calculate the average accuracy of image-text matching in multi-object scenarios.

[0010] Furthermore, the joint hash code matrix learning formula is:

[0011]

[0012] stE1, E2 ≥ 0, B∈{-1, 1} r×n ,

[0013] in, is the multi-object image data matrix, is a multi-object text data matrix, where d1 and d2 are the dimensions of multi-object image and text data, and n is the number of samples. is the multi-object image and text data matrix to be processed by kernelization, where q is the dimension of the kernelized data. For the column vector x at a certain position in the multi-object image and text data matrix, the column vector at the corresponding position in the kernelized multi-object image and text matrix is in is the square of the 2-norm of the vector, is the q column vectors randomly extracted from the corresponding data matrix without duplication, q is also the dimension of the kernelized data, is the kernel width, x i is the i-th column vector of the corresponding data matrix, g j for The jth column vector in R, ||·||2 is the 2-norm of the vector, U1, U2∈R q×r are the latent factor matrices of multi-object images and text data, respectively, where r is the hash code length, L∈R c×n is the category labeling matrix. When the jth data contains the i-th object, L ij =1, otherwise L ij = 0, where c is the number of object types, is the normalized category label matrix, l i is the i-th column vector of the category label matrix, Γ∈R c×n is the index matrix, when L ij =1, Γ ij =1, otherwise Γ ij =-1, ⊙ is the Hadamard product symbol, E1, E2∈R c×n are the interval element matrices of multi-object images and text data, respectively, which are used to strengthen the constraint of heterogeneous data being far away, P1, P2∈R c×r are the information projection matrices of multi-object images and text data, V1, V2∈R r×n is an independent latent representation space for multi-object images and texts, For heterogeneous information interaction, Used to guide heterogeneous information interaction, Q1, Q2∈R r×r are projection matrices with orthogonal constraints for multi-object images and text data, B∈{-1,1} r×n is a joint hash code matrix whose elements are -1 or 1, is the square of the F-norm of the matrix, I r is the identity matrix of dimension r, and λ1, λ2, α, β1, β2, and μ are trade-off parameters.

[0014] Furthermore, the alternating update algorithm 1 is summarized as follows:

[0015]

[0016]

[0017] Furthermore, the hash function learning formula is:

[0018]

[0019] stB∈{-1,1} r×n

[0020] in, is the normalized category label matrix, F1, F2∈R r×q are the hash functions for multi-object images and text data, respectively, where r is the hash code length, q is the dimension of the kernelized data mentioned in the joint hash code matrix learning formula, and μ1 and μ2 are trade-off parameters.

[0021] Furthermore, the alternating update algorithm 2 is summarized as follows:

[0022]

[0023] The present invention provides a method for image-text matching in a multi-object scenario based on heterogeneous information interaction, which has the following advantages:

[0024] (1) The method learns the independent latent representation spaces of images and texts in the joint hash code matrix learning formula, retaining the unique information of images and texts, thereby making the latent representation space more discriminative.

[0025] (2) The method uses heterogeneous information interaction in the joint hash code matrix learning formula to link the independent latent representation spaces of images and texts, making the latent representations of the two modalities closer. At the same time, it can integrate multi-object information into the latent representation of each modality. With the help of the latent representations of the above modalities, a more discriminative joint hash code matrix B can be learned.

[0026] (3) The method imposes heterogeneous information interaction constraints on the hash function in the hash function learning formula, so that the encoding features obtained by hash functions F1 and F2 when projecting images and texts into the Hamming space are closer to the joint hash code matrix B, thereby improving the matching accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 This is a flow chart of the image-text matching method in a multi-object scenario based on heterogeneous information interaction provided by the present invention; DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0030] Exemplary Methods

[0031] like Figure 1 The present invention provides a method for image-text matching in a multi-object scenario based on heterogeneous information interaction, and the method steps are as follows:

[0032] Step S110: Divide the data set into a training set and a test set.

[0033] Step S120: Constructing a joint hash code matrix learning formula for the multi-object data graph-text matching problem.

[0034] The joint hash code matrix learning formula is as follows:

[0035]

[0036] stE1, E2 ≥ 0, B∈{-1, 1} r×n ,

[0037] in, is the multi-object image data matrix, is a multi-object text data matrix, where d1 and d2 are the dimensions of multi-object image and text data, and n is the number of samples. is the multi-object image and text data matrix to be processed by kernelization, where q is the dimension of the kernelized data. For the column vector x at a certain position in the multi-object image and text data matrix, the column vector at the corresponding position in the kernelized multi-object image and text matrix is in is the square of the 2-norm of the vector, is the q column vectors randomly extracted from the corresponding data matrix without duplication, q is also the dimension of the kernelized data, is the kernel width, x i is the i-th column vector of the corresponding data matrix, g j for The jth column vector in R, ||·||2 is the 2-norm of the vector, U1, U2∈R q×r are the latent factor matrices of multi-object images and text data, respectively, where r is the hash code length, L∈R c×n is the category labeling matrix. When the jth data contains the i-th object, L ij =1, otherwise L ij = 0, where c is the number of object types, is the normalized category label matrix, l i is the i-th column vector of the category label matrix, Γ∈R c×n is the index matrix, when L ij =1, Γ ij =1, otherwise Γ ij =-1, ⊙ is the Hadamard product symbol, E1, E2∈R c×n are the interval element matrices of multi-object images and text data, respectively, which are used to strengthen the constraint of heterogeneous data being far away, P1, P2∈R c×r are the information projection matrices of multi-object images and text data, V1, V2∈R r×n is an independent latent representation space for multi-object images and texts, For heterogeneous information interaction, Used to guide heterogeneous information interaction, Q1, Q2∈R r×r are projection matrices with orthogonal constraints for multi-object images and text data, B∈{-1,1} r×n is a joint hash code matrix whose elements are -1 or 1, is the square of the F-norm of the matrix, I r is the identity matrix of dimension r, and λ1, λ2, α, β1, β2, and μ are trade-off parameters.

[0038] Step S130: Solve the joint hash code matrix learning formula using the alternating update algorithm 1 to obtain the joint hash code matrix B.

[0039] The alternating update algorithm 1 is summarized as follows:

[0040]

[0041]

[0042] Step S140: Construct a hash function learning formula for the multi-object data graph-text matching problem.

[0043] The hash function learning formula is as follows:

[0044]

[0045] stB∈{-1,1} r×n

[0046] in, is the normalized category label matrix, F1, F2∈R r×q are the hash functions for multi-object images and text data, respectively, where r is the hash code length, q is the dimension of the kernelized data mentioned in the joint hash code matrix learning formula, and μ1 and μ2 are trade-off parameters.

[0047] Step S150: Solve the hash function learning formula using the alternating update algorithm 2 to obtain hash functions F1 and F2.

[0048] The alternating update algorithm 2 is summarized as follows:

[0049]

[0050] Step S160: Use the joint hash code matrix B and hash functions F1 and F2 obtained in the above steps to test the test set and calculate the average accuracy of image-text matching in the multi-object scene.

[0051] Through this implementation, the data set is first divided into a training set and a test set, and then a joint hash code matrix learning formula for the multi-object data image-text matching problem is constructed. The joint hash code matrix learning formula is solved using the alternating update algorithm 1 to obtain the joint hash code matrix B. Then, a hash function learning formula for the multi-object data image-text matching problem is constructed, and the hash function learning formula is solved using the alternating update algorithm 2 to obtain hash functions F1 and F2. Finally, the joint hash code matrix B and hash functions F1 and F2 obtained in the above steps are used to test the test set and calculate the average accuracy of image-text matching in the multi-object scenario.

[0052] To further illustrate, assuming that a multi-object image-text matching dataset is subjected to image-text matching according to this embodiment, an average accuracy rate higher than that of most methods will be obtained.

[0053] Detailed description of the invention Results

[0054] This implementation uses a publicly available multi-object image-text matching dataset, the details of which are described as follows:

[0055] The COCO dataset consists of 122,218 text-image pairs, covering 80 categories. These text-image pairs belong to at least one of the categories. The images are represented by 4096-dimensional features extracted by the Caffe framework, and the texts are represented by 2000-dimensional features extracted using the bag-of-words model.

[0056] To verify the superiority of this implementation (Ours), we compare this implementation with several existing image-text matching methods, including SCRATCH, LEMON, BATCH, and HCCH. We compare the mean average precision (mAP) of these methods for image-text matching on the COCO dataset using hash codes of different lengths. The specific data comparison is shown in Table 1.

[0057] Table 1 Average accuracy mAP (%) of image-text matching in COCO dataset

[0058] Hash code length SCRATCH LEMON BATCH HCCH Ours 16 50.32 52.08 42.66 57.05 58.24 32 54.01 52.56 43.70 57.22 59.49 64 51.76 53.29 43.92 57.27 60.56 128 47.69 53.30 44.50 57.89 60.58

[0059] Comparing the data in the two tables above, we can see that Ours achieves the best performance for each selected hash code length, significantly improving the average accuracy of image-text matching in multi-object scenarios. Quantitative results fully demonstrate the superiority of Ours, as Ours preserves the unique information of image and text modalities; Ours uses heterogeneous information interaction to communicate the latent representation spaces of images and text, promoting their alignment and integrating multi-object information into the latent representation space to better solve the image-text matching problem in multi-object scenarios; the latent representation space learned using these two strategies is more discriminative, facilitating the generation of a high-quality joint hash code matrix; Ours imposes heterogeneous information interaction constraints on the hash function learning formula, making the encoded features obtained by the hash function projecting feature data into the Hamming space closer to the joint hash code matrix. Extensive experiments demonstrate that this method outperforms most existing methods. The parameters of this embodiment are set as follows: q=1500, λ1=0.7, λ2=0.3, α=20, β1=0.45, β2=0.55, μ=5e-6, μ1=3e-6, μ2=5e-6, η=5e-6.

[0060] This embodiment proposes a method for image-text matching in multi-object scenarios based on heterogeneous information interaction. This method is used for image-text matching in multi-object scenarios. It learns the latent representation spaces of images and text separately to retain modality-specific information. It also uses heterogeneous information interaction to embed multi-object information into the latent representation space to solve the image-text matching problem in multi-object scenarios. The alternating update algorithm 1 is used to solve the constructed joint hash code matrix learning formula to obtain the joint hash code matrix B. The alternating update algorithm 2 is then used to solve the constructed hash function learning formula to obtain hash functions F1 and F2. Finally, the joint hash code matrix B and hash functions F1 and F2 are used to test the test set and calculate the average accuracy of image-text matching in multi-object scenarios. Experimental results on the public dataset COCO show that this embodiment has a higher average matching accuracy than other methods and is superior.

[0061] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or illustrative of the principles of the present invention and do not constitute limitations of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included within the scope of protection of the present invention. In addition, the appended claims are intended to cover all variations and modifications that fall within the scope and metes and bounds of the appended claims, or equivalents thereof.

Claims

1. A method for image-text matching in a multi-object scenario based on heterogeneous information interaction, the method comprising the steps of: Divide the dataset into training set and test set; Construct a joint hash code matrix learning formula for the multi-object data image-text matching problem; The joint hash code matrix learning formula is: in, is the multi-object image data matrix, is a multi-object text data matrix, where d1 and d2 are the dimensions of multi-object image and text data, and n is the number of samples. is the multi-object image and text data matrix to be processed by kernelization, where q is the dimension of the kernelized data. For the column vector x at a certain position in the multi-object image and text data matrix, the column vector at the corresponding position in the kernelized multi-object image and text matrix is in is the square of the 2-norm of the vector, is the q column vectors randomly extracted from the corresponding data matrix without duplication, q is also the dimension of the kernelized data, is the kernel width, x i is the i-th column vector of the corresponding data matrix, g j for The jth column vector in R, ||·||2 is the 2-norm of the vector, U1, U2∈R q×r are the latent factor matrices of multi-object images and text data, respectively, where r is the hash code length, L∈R c×n is the category labeling matrix. When the jth data contains the i-th object, L ij =1, otherwise L ij = 0, where c is the number of object types, is the normalized category label matrix, l i is the i-th column vector of the category label matrix, Γ∈R c×n is the index matrix, when L ij =1, Γ ij =1, otherwise Γ ij =-1, ⊙ is the Hadamard product symbol, E1, E2∈R c×n are the interval element matrices of multi-object images and text data, respectively, which are used to strengthen the constraint of heterogeneous data being far away, P1, P2∈R c×r are the information projection matrices of multi-object images and text data, V1, V2∈R r×n V1 is an independent latent representation space for multi-object images and texts. T V2 is used for heterogeneous information interaction. Used to guide heterogeneous information interaction, Q1, Q2∈R r×r are projection matrices with orthogonal constraints for multi-object images and text data, B∈{-1,1} r×n is a joint hash code matrix whose elements are -1 or 1, is the square of the F-norm of the matrix, I r is the identity matrix of dimension r, λ1, λ2, α, β1, β2, μ are trade-off parameters; Use the alternating update algorithm 1 to solve the joint hash code matrix learning formula and obtain the joint hash code matrix B; The alternating update algorithm 1 is summarized as follows: Construct a hash function learning formula for multi-object data graph-text matching problem; The hash function learning formula is: in, is the normalized category label matrix, F1, F2∈R r×q are the hash functions for multi-object images and text data, respectively, where r is the hash code length, q is the dimension of the kernelized data mentioned in the joint hash code matrix learning formula, and μ1 and μ2 are trade-off parameters; Use the alternating update algorithm 2 to solve the hash function learning formula and obtain hash functions F1 and F2; The alternating update algorithm 2 is summarized as follows: The joint hash code matrix B and hash functions F1 and F2 obtained in the above steps are used to test the test set and calculate the average accuracy of image-text matching in multi-object scenarios.

Images