Text image matching method and system based on large language model entropy increase

Abstract

The invention relates to the technical field of computer vision, and provides a text image matching method and system based on large language model entropy increase, and the method comprises the steps: enabling data set image features to pass through a visual attention encoder to obtain original image codes; constructing a hypergraph based on original image coding and generating hypergraph image features through a hypergraph neural network; performing residual connection on the hypergraph image features and original image codes to construct a visual hypergraph adapter; inducing a large language model to generate a synonym material through a prompt learning method by utilizing a first text feature generated by the image feature description, and obtaining an entropy-increased fusion text feature; encoding the fused text feature and the first text feature; constructing a hypergraph based on multi-dimensional feature coding, and generating hypergraph text features through a hypergraph neural network; performing dimension reduction on the supergraph text features, and then combining with original corpus feature codes to construct a text supergraph adapter; and calculating a similarity matrix based on the visual hypergraph adapters and the text hypergraph adapters to obtain a similarity ranking.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to a text image matching method and system based on large language model entropy enhancement. Background Art

[0002] Text-image matching based on visual semantic embedding is a general solution for image-text matching. Its core idea is to globally associate correspondences by projecting images and texts into a common space using two separate networks. Subsequent work seeks better representative common subspaces, designs more appropriate similarity metrics, and proposes visual language pre-training methods.

[0003] At present, an image-text matching method that integrates local and global semantics has been proposed. The image and text features are extracted respectively through feature extraction algorithms; the relationship between local features in the image is further mined through the visual context encoding module to obtain contextual relationships and more semantic information, and then the global and local similarities of the image and text are calculated; finally, the global and local image-text similarities are integrated to form a multi-granularity fusion network to improve the accuracy of image-text matching. Although the algorithm integrates local and global semantics, for the two modalities of image and text, there is an entropy difference between the text modality and the visual modality, which makes it difficult for the matching algorithm to accurately capture the relationship between the two, and it is easy to produce deviations when processing information, which leads to the accuracy of the matching results not meeting the current application requirements. Summary of the invention

[0004] In order to overcome the defect of low matching accuracy caused by the entropy difference between the text modality and the visual modality in the above-mentioned prior art, the present invention provides a text image matching method and system based on large language model entropy increase.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A text image matching method based on large language model entropy increase includes the following steps:

[0007] Collecting an image data set and extracting image features of the data set, and encoding the image features of the data set through a visual attention encoder to obtain an original image encoding;

[0008] Encode the original image to construct a hypergraph using a k-nearest neighbor method, and use a hypergraph neural network to construct a multilateral connection relationship to generate a hypergraph image feature;

[0009] Perform residual concatenation of the hypergraph image features and the original image code to construct a visual hypergraph adapter;

[0010] Generate a first text feature according to the image feature description extracted from the image data set, input the first text feature into a large language model, induce the large language model to generate a plurality of synonymous corpora as second text features after entropy increase through a prompt learning method, and splice the first text feature with the second text feature to obtain a multi-dimensional fusion text feature;

[0011] The fused text feature and the first text feature are respectively passed through a text encoder to generate a multidimensional feature code and an original corpus feature code;

[0012] The multi-dimensional feature encoding is used to construct a hypergraph by a k-nearest neighbor method and a hypergraph neural network is used to construct a multilateral connection relationship to generate a hypergraph text feature;

[0013] After reducing the dimension of the hypergraph text feature to the original dimension, the residual linking structure is performed with the original corpus feature coding to construct a text hypergraph adapter;

[0014] After aligning the dimensions of the visual hypergraph adapter and the textual hypergraph adapter, a similarity matrix is ​​calculated to obtain a similarity ranking and output it as a matching result.

[0015] Furthermore, the present invention also proposes a text image matching system based on large language model entropy increase, and applies the text image matching method proposed in the present invention. The system comprises:

[0016] A data acquisition module, used for acquiring image data sets;

[0017] A feature extraction module is used to extract features from an image data set to obtain image features and image feature descriptions of the data set;

[0018] A visual hypergraph adapter module is used to construct a hypergraph by encoding the original image in a k-nearest neighbor manner, and use a hypergraph neural network to construct a multilateral connection relationship to generate a hypergraph image feature; and perform a residual connection structure between the hypergraph image feature and the original image encoding to construct a visual hypergraph adapter;

[0019] A text modality entropy increasing module, for generating a first text feature according to an image feature description extracted from the image data set, inputting the first text feature into a large language model, inducing the large language model to generate a plurality of synonymous corpora as second text features after entropy increase through a prompt learning method, and concatenating the first text feature with the second text feature to obtain a multi-dimensional fused text feature;

[0020] A text encoding module, used for respectively generating a multi-dimensional feature encoding and an original corpus feature encoding by using a text encoder to encode the fused text feature and the first text feature;

[0021] A text hypergraph adapter module is used to construct a hypergraph by using a k-nearest neighbor method to encode the multidimensional features and use a hypergraph neural network to construct a multilateral connection relationship to generate a hypergraph text feature; after reducing the dimension of the hypergraph text feature to the original dimension, the residual linking structure is performed with the original corpus feature encoding to construct a text hypergraph adapter;

[0022] The matching module is used to calculate the similarity matrix after aligning the dimensions of the visual hypergraph adapter and the text hypergraph adapter, obtain the similarity ranking and output it as the matching result.

[0023] Furthermore, the present invention also proposes a device, including a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor executes all or part of the steps of the text image matching method described in the present invention.

[0024] Furthermore, the present invention also proposes a storage medium on which computer-readable instructions are stored, wherein the computer-readable instructions, when executed by a processor, implement all or part of the steps of the text image matching method described in the present invention.

[0025] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0026] To address the problem of the modal entropy difference between the textual modality and the visual modality, the present invention uses a prompt learning method to induce a large model to output an open corpus with synonymous understanding, thereby increasing the relatively correct information entropy of the textual modality and reducing the entropy difference between the modalities. To address the problem of sparse matching of images and texts, the present invention uses an open corpus with synonymous understanding to guide the hypergraph adapter and uses residual connections to guide the updating of the original features in the embedding space, thereby correcting the distortion of the embedding space caused by sparse matching, generating a more accurate image-text matching ranking, and effectively improving the matching accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a flow chart of a text image matching method according to an embodiment of the present invention.

[0028] Figure 2 A framework diagram of a text image matching method according to an embodiment of the present invention.

[0029] Figure 3 The figure is a schematic diagram of image-text retrieval matching results according to an embodiment of the present invention.

[0030] Figure 4 The figure is a schematic diagram of text-image retrieval matching results according to an embodiment of the present invention.

[0031] Figure 5The figure is an architecture diagram of a text image matching system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0033] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0035] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Example 1

[0037] This embodiment proposes a text image matching method based on large language model entropy increase. Figure 1 , 2 , which is a flow chart of the text image matching method based on large language model entropy increase in this embodiment.

[0038] The text image matching method based on large language model entropy increase proposed in this embodiment includes the following steps:

[0039] S1, collect image data sets and extract image features of the data sets, and obtain original image encoding by passing the image features of the data sets through a visual attention encoder;

[0040] S2, encoding the original image to construct a hypergraph using a k-nearest neighbor method, and using a hypergraph neural network to construct a multilateral connection relationship to generate a hypergraph image feature;

[0041] S3, performing residual concatenation of the hypergraph image features and the original image coding to construct a visual hypergraph adapter;

[0042] S4, generating a first text feature according to the image feature description extracted from the image data set, inputting the first text feature into a large language model, inducing the large language model to generate a plurality of synonymous corpora as second text features after entropy increase through a prompt learning method, and splicing the first text feature with the second text feature to obtain a multi-dimensional fusion text feature;

[0043] S5, generating a multidimensional feature code and an original corpus feature code by respectively using a text encoder for the fused text feature and the first text feature;

[0044] S6, encoding the multidimensional features to construct a hypergraph by a k-nearest neighbor method, and using a hypergraph neural network to construct a multilateral connection relationship to generate a hypergraph text feature;

[0045] S7, reducing the dimension of the hypergraph text feature to the original dimension and then performing residual linking with the original corpus feature coding to construct a text hypergraph adapter;

[0046] S8. Dimensionally aligning the visual hypergraph adapter and the text hypergraph adapter, calculate a similarity matrix, obtain a similarity ranking and output it as a matching result.

[0047] In this embodiment, the problem of the difference in modal entropy between the textual modality and the visual modality is addressed. A prompt learning method is used to induce a large model to output an open corpus with synonymous understanding, thereby increasing the relatively correct information entropy of the textual modality and reducing the entropy difference between the modalities. In order to address the problem of sparse matching of images and texts, the hypergraph adapter is guided by using open corpus with synonymous understanding and residual connections are used to guide the updating of the original features in the embedding space, thereby correcting the distortion of the embedding space caused by sparse matching and generating a more accurate image-text matching ranking.

[0048] In an optional embodiment, in step S1, Fast-Rcnn is used to extract image features from the collected image data set.

[0049] In an optional embodiment, the visual attention encoder in step S1 includes a fully connected layer and a multi-head self-attention module, wherein the fully connected layer is used to capture feature connection relationships; and the multi-head self-attention module is used to learn complex relationships between features.

[0050] In this embodiment, the multi-head self-attention mechanism is integrated to capture features of different scales, so that the model can pay attention to local details and global context at the same time, enhancing the understanding and encoding capabilities of visual information.

[0051] In an optional embodiment, in step S2, in the process of encoding the original image to construct a hypergraph by a k-nearest neighbor method, the following steps are included:

[0052] Standardize the original image encoding;

[0053] Based on the preset K value, for each image node, use the KNN algorithm to find its K nearest neighbor nodes;

[0054] For each node, it and its K nearest neighbors form a hyperedge, where each hyperedge can contain a central node and its K neighbor nodes;

[0055] Create an incidence matrix H whose size is the number of nodes times the number of hyperedges;

[0056] Assign a weight to each hyperedge, and then combine the node set, hyperedge set, and weights to obtain a hypergraph.

[0057] The hypergraph described in this embodiment captures high-order relationships between image features.

[0058] Furthermore, we use Hypergraph Neural Networks (HGNN) to learn the features of the hypergraph to construct a multilateral connection relationship and obtain the hypergraph image feature P hgnn .

[0059] Furthermore, in step S3, the hypergraph image feature P hgnn Perform residual joint structure with the original image code P to construct the visual hypergraph adapter P final ; Its expression is:

[0060] P final =αP+(1-α)P hgnn

[0061] Wherein, α∈[0,1] is a weight coefficient. In this embodiment, the feature embedding space is corrected by the multilateral connection formed by the hypergraph network to prevent the network from overfitting and mode update direction deviation.

[0062] In an optional embodiment, the steps in step S4 include:

[0063] S4.1. According to the image feature description extracted from the image dataset, an attention score is obtained based on top-down attention calculation, and a first text feature T is obtained by arrangement. dateset ={t i |i∈[1,L],t i ∈R l}; where t irepresents the i-th text feature word; l is the dimension size of the first text feature; L is the number of extracted image feature descriptions;

[0064] S4.2, construct a prompt sentence template, and transform the first text feature T dateset Fill in the prompt sentence template and use it as the input of the large language model to induce the large language model to generate corresponding synonymous corpus as the second text feature T after entropy increase synonyms ={t j |j∈[1,L],t j ∈R d}; where t j represents the jth text feature word; d is the dimension size of the second text feature;

[0065] S4.3, the first text feature T dateset and the second text feature T synonyms Concatenate rows into a multi-dimensional fusion text feature F, where F∈R d+l .

[0066] Optionally, Fast-RCNN is used to extract image feature descriptions from the collected image dataset through top-down attention.

[0067] The prompt sentence template can be set according to the application scenario, for example, such as "{}, Generate synonymous sentences", and then the first text feature T dateset After filling in "{}", it is used as the input of the large language model, and the understanding ability of the large model is further utilized to induce the large language model to generate multiple synonymous corpora.

[0068] Exemplarily, the large language model is Llama-3-8B, which contains a vocabulary of 128K and uses more than 15 trillion labeled public data for training, supporting a wide range of application scenarios.

[0069] In an optional embodiment, the text encoder includes an embedding module, a context understanding model and a fully connected layer. The embedding module is used to map word units into dense vectors; the context understanding model is used to understand the context sequence of the dense vectors; and the fully connected layer is used to integrate features and output text feature encoding.

[0070] Exemplarily, the context understanding model adopts Bigru or Bert, wherein Bigru is a bidirectional gated recurrent unit and Bert is a bidirectional encoder representation.

[0071] In an optional embodiment, in step S7, reducing the dimension of the hypergraph text feature to the original dimension and performing residual connection with the original corpus feature code comprises the following steps:

[0072] By cutting down the hypergraph text feature to its original dimension, we can get the multi-edge connection feature code F′ Dimen-hgnn ;

[0073] Encode the multi-edge connection feature F′ Dimen-hgnn and the original corpus feature encoding F ori Perform residual connection to obtain the text hypergraph adapter F final ; Its expression is:

[0074] F final =βF ori +(1-β)F′ Dimen-hgnn

[0075] Among them, β∈[0,1] is the weight coefficient; F′ Dimen-hgnn represents the multi-edge connection feature encoding, F ori Represents the original corpus feature encoding.

[0076] Among them, by reducing the hypergraph text features to the original dimensions, the original feature data is attached with the large model semantic information entropy weight, realizing the fusion of open semantics and original semantics.

[0077] Furthermore, the multilateral connection feature encoding F′ Dimen-hgnn and the original corpus feature encoding F ori Perform residual connection to correct the offset of the original embedding space position, that is, through the multi-edge connection feature encoding F′ Dimen-hgnn Improve the original data F ori , and correct its matching error in the synonymous embedding space.

[0078] In an optional embodiment, step S8 includes the following steps:

[0079] Aligning the dimensions of the visual hypergraph adapter and the textual hypergraph adapter through a mask operation, and then multiplying the feature matrices of the two to obtain a similarity matrix;

[0080] The similarity matrix is ​​smoothed by LSEpooling, and a similarity ranking is obtained according to the ranking of the matrix elements, and a matching result is output according to the similarity ranking.

[0081] In this embodiment, taking into account the variable-length sequence characteristics of bimodal data, the modal matrix dimensions are aligned through a mask operation, and the feature matrices are multiplied to form a similarity matrix. The noise caused by the mask operation is removed and the data is smoothed through LSEpooling. The similarity ranking is calculated based on the size of the product, and the top n results are output as the text image matching results.

[0082] For example, in a specific implementation process, the text image matching method proposed in this embodiment is applied to the COCO dataset, and the following is obtained: Figure 3 , 4 The matching results are shown.

[0083] Among them, in Figure 3 In the image-text retrieval shown, the results obtained by using the traditional image-text matching algorithm include more than two completely disordered sentences that are not true values, and their texts have no keyword association with the image content; while the retrieval results of the matching algorithm proposed in this embodiment are consistent with the image content, and even the second retrieval result, which has the largest gap with the image description, can retrieve synonyms such as "suitcase".

[0084] In such Figure 4 In the text-image retrieval shown, sentence information irrelevant to the image target in the results obtained by using the traditional image-text matching algorithm is incorrectly aggregated and retrieved; the matching algorithm proposed in this embodiment can retrieve more true values ​​of adaptive results, and the erroneous results can also retrieve images close to the text description and calibrate the position of the embedding space.

[0085] The methods proposed in this embodiment have shown significant superiority in the five-fold cross-validation and direct validation performed on the COCO 5K test set. Especially on the larger and more complex MS-COCO dataset, most methods still maintain a certain degree of accuracy in the five-fold cross-validation. However, on the 5K complete test set, the performance of strategies that only rely on encoder method optimization or methods to improve the backbone network decreases significantly as the amount of data increases. In contrast, the statements processed by hypergraph modeling and open semantic entropy increase significantly reduce the confusion of the embedding space, so that the retrieval results of the present invention in R@5 are almost the same as the true value, and the retrieval accuracy rate of more than 80% can also be achieved on R@1.

[0086] The method proposed in this embodiment is applied to the Flickr30K test set. From its quantitative results, it can be seen that compared with the baseline model CHAN, this embodiment is divided into two schemes, BiGRU-based and Bert-based, according to the different encoders used. Among them, the BiGRU-based scheme has achieved a significant improvement of more than 16.8% and 40.1% in R@1 of bidirectional retrieval; and the Bert-based scheme has performed 14% better than the most advanced technology on RSUM, showing overwhelming performance superiority. In addition, the performance difference of image-text mutual retrieval has also been greatly improved. The performance of text-image retrieval after entropy increase is no longer significantly behind that of image-text retrieval. Based on BiGRU, the performance gap of image-text mutual retrieval is reduced from 32% to 10.7%; and based on Bert, it is further reduced to only 2%, which proves the effectiveness of the modal entropy increase method in improving mutual retrieval performance.

[0087] Example 2

[0088] This embodiment proposes a text image matching system based on large language model entropy increase, and applies the text image matching method proposed in Embodiment 1. Figure 5 The figure shows the architecture diagram of the text image matching system of this embodiment.

[0089] The text image matching system based on large language model entropy increase proposed in this embodiment includes:

[0090] A data acquisition module, used for acquiring image data sets;

[0091] A feature extraction module is used to extract features from an image data set to obtain image features and image feature descriptions of the data set;

[0092] A visual hypergraph adapter module is used to construct a hypergraph by encoding the original image in a k-nearest neighbor manner, and use a hypergraph neural network to construct a multilateral connection relationship to generate a hypergraph image feature; and perform a residual connection structure between the hypergraph image feature and the original image encoding to construct a visual hypergraph adapter;

[0093] A text modality entropy increasing module, for generating a first text feature according to an image feature description extracted from the image data set, inputting the first text feature into a large language model, inducing the large language model to generate a plurality of synonymous corpora as second text features after entropy increase through a prompt learning method, and concatenating the first text feature with the second text feature to obtain a multi-dimensional fused text feature;

[0094] A text encoding module, used for respectively generating a multi-dimensional feature encoding and an original corpus feature encoding by using a text encoder to encode the fused text feature and the first text feature;

[0095] A text hypergraph adapter module is used to construct a hypergraph by using a k-nearest neighbor method to encode the multidimensional features and use a hypergraph neural network to construct a multilateral connection relationship to generate a hypergraph text feature; after reducing the dimension of the hypergraph text feature to the original dimension, the residual linking structure is performed with the original corpus feature encoding to construct a text hypergraph adapter;

[0096] The matching module is used to calculate the similarity matrix after aligning the dimensions of the visual hypergraph adapter and the text hypergraph adapter, obtain the similarity ranking and output it as the matching result.

[0097] It can be understood that the system of this embodiment corresponds to the method of the above-mentioned embodiment 1, and the options in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described repeatedly here.

[0098] Example 3

[0099] This embodiment proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes all or part of the steps of the text image matching method proposed in Example 1.

[0100] Example 4

[0101] This embodiment proposes a storage medium on which computer-readable instructions are stored, wherein the computer-readable instructions, when executed by a processor, implement all or part of the steps of the text image matching method proposed in Embodiment 1.

[0102] Exemplarily, the storage medium includes, but is not limited to, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0103] Exemplarily, the instructions, programs, code sets or instruction sets may be implemented using conventional programming languages.

[0104] Exemplarily, the processor includes but is not limited to a smart phone, a personal computer, a server, a network device, etc., and is used to execute all or part of the steps of the text image matching method described in Example 1.

[0105] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely exemplary, in which the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same or one or more software and / or hardware when implementing the scheme of the present invention. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the scheme of this embodiment.

[0106] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A text image matching method based on large language model entropy increase, characterized in that: The following steps are involved: Collecting an image data set and extracting image features of the data set, and encoding the image features of the data set through a visual attention encoder to obtain an original image encoding; Encode the original image to construct a hypergraph using a k-nearest neighbor method, and use a hypergraph neural network to construct a multilateral connection relationship to generate a hypergraph image feature; Perform residual concatenation of the hypergraph image features and the original image code to construct a visual hypergraph adapter; Generate a first text feature according to the image feature description extracted from the image data set, input the first text feature into a large language model, induce the large language model to generate a plurality of synonymous corpora as second text features after entropy increase through a prompt learning method, and splice the first text feature with the second text feature to obtain a multi-dimensional fusion text feature; The fused text feature and the first text feature are respectively passed through a text encoder to generate a multidimensional feature code and an original corpus feature code; The multi-dimensional feature encoding is used to construct a hypergraph by a k-nearest neighbor method and a hypergraph neural network is used to construct a multilateral connection relationship to generate a hypergraph text feature; After reducing the dimension of the hypergraph text feature to the original dimension, the residual linking structure is performed with the original corpus feature coding to construct a text hypergraph adapter; After aligning the dimensions of the visual hypergraph adapter and the textual hypergraph adapter, a similarity matrix is ​​calculated to obtain a similarity ranking and output it as a matching result.

2. The text image matching method according to claim 1, characterized in that: The visual attention encoder includes a fully connected layer and a multi-head self-attention module, wherein the fully connected layer is used to capture feature connection relationships; the multi-head self-attention module is used to learn complex relationships between features.

3. The text image matching method according to claim 1, characterized in that: The first text feature is input into a large language model, a prompt learning method is used to induce the large language model to generate a plurality of synonymous corpora as second text features after entropy increase, and the first text feature and the second text feature are spliced ​​to obtain a multi-dimensional fusion text feature, including the following steps: According to the image feature description extracted from the image dataset, an attention score is obtained based on top-down attention calculation, and the first text feature T is compiled. daghet ={t i |i∈[1,L],t i ∈R l }; where t i represents the i-th text feature word; l is the dimension size of the first text feature; L is the number of extracted image feature descriptions; Construct a prompt sentence template and transform the first text feature T ddteset Fill in the prompt sentence template and use it as the input of the large language model to induce the large language model to generate corresponding synonymous corpus as the second text feature T after entropy increase synonyms ={t j |j∈[1,L],t j ∈R d }; where t j represents the jth text feature word; d is the dimension size of the second text feature; The first text feature T dateset and the second text feature T synonyms Concatenate rows into a multi-dimensional fusion text feature F, where F∈R d+l .

4. The text image matching method according to claim 1, characterized in that: The text encoder comprises: Embedding module, used to map word units into dense vectors; Contextual understanding models for contextual sequence understanding of dense vectors; The fully connected layer is used to integrate features and output text feature encoding.

5. The text image matching method according to claim 1, characterized in that: The step of reducing the dimension of the hypergraph text feature to the original dimension and performing residual connection with the original corpus feature code comprises the following steps: Reducing the dimension of the hypergraph text feature to the original dimension by interception to obtain a multilateral connection feature encoding; The multi-edge connection feature code is residually connected with the original corpus feature code to obtain the text hypergraph adapter F final ; Its expression is: F final =βF ori +(1-β)F′ Dimen-hgnn Among them, β∈[0,1] is the weight coefficient; F′ Dimrn-hgnn represents the multi-edge connection feature encoding, F ori Represents the original corpus feature encoding.

6. The text image matching method according to claim 1, characterized in that: The step of calculating a similarity matrix after aligning the dimensions of the visual hypergraph adapter and the textual hypergraph adapter comprises the following steps: Aligning the dimensions of the visual hypergraph adapter and the textual hypergraph adapter through a mask operation, and then multiplying the feature matrices of the two to obtain a similarity matrix; The similarity matrix is ​​smoothed by LSEpooling, and a similarity ranking is obtained according to the ranking of the matrix elements, and a matching result is output according to the similarity ranking.

7. The text image matching method according to any one of claims 1 to 6, characterized in that: The method further comprises the following steps: The collected image data set is subjected to feature extraction through a feature extraction model based on Fast-Rcnn to obtain image features of the data set and image feature description.

8. A text image matching system based on large language model entropy increase, applying the text image matching method according to any one of claims 1 to 7, characterized in that: include: A data acquisition module, used for acquiring image data sets; A feature extraction module is used to extract features from an image data set to obtain image features and image feature descriptions of the data set; A visual hypergraph adapter module is used to construct a hypergraph by encoding the original image in a k-nearest neighbor manner, and use a hypergraph neural network to construct a multilateral connection relationship to generate a hypergraph image feature; and perform a residual connection structure between the hypergraph image feature and the original image encoding to construct a visual hypergraph adapter; A text modality entropy increasing module, for generating a first text feature according to an image feature description extracted from the image data set, inputting the first text feature into a large language model, inducing the large language model to generate a plurality of synonymous corpora as second text features after entropy increase through a prompt learning method, and concatenating the first text feature with the second text feature to obtain a multi-dimensional fused text feature; A text encoding module, used for respectively generating a multi-dimensional feature encoding and an original corpus feature encoding by using a text encoder to encode the fused text feature and the first text feature; A text hypergraph adapter module is used to construct a hypergraph by using a k-nearest neighbor method to encode the multidimensional features and use a hypergraph neural network to construct a multilateral connection relationship to generate a hypergraph text feature; after reducing the dimension of the hypergraph text feature to the original dimension, the residual linking structure is performed with the original corpus feature encoding to construct a text hypergraph adapter; The matching module is used to calculate the similarity matrix after aligning the dimensions of the visual hypergraph adapter and the text hypergraph adapter, obtain the similarity ranking and output it as the matching result.

9. A device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the computer-readable instructions are executed by the processor, the processor executes all or part of the steps of the text image matching method according to any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, all or part of the steps of the text image matching method according to any one of claims 1 to 7 are implemented.

Images