Image-text retrieval method and system based on big data
By calculating the graphic and text modal fusion entropy in graphic and text retrieval and performing attention conversion, obtaining the fusion representation of image and text features, the retrieval difficulty problem caused by heterogeneity of image and text data is solved, and more efficient and robust Image and text search effect.
Patent Information
- Application Number
- CN202411437690.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-15
- Publication Date
- 2025-02-07
AI Technical Summary
In the prior art, the heterogeneity of image data and text data makes feature extraction and fusion complex, increasing the difficulty of graphic and text retrieval. How to effectively modal fusion image modal information and text modal information to improve retrieval performance is A difficult problem.
By obtaining the graphic and text modal data to be retrieved, feature extraction is performed on the image and text data, calculating the graphic and text modal fusion entropy between the image modal information and text modal information, and determining the corresponding image feature representation based on the entropy value The text description array and text feature representation correspond to the image description array. Careful conversion is performed to obtain local and global graphic and text fusion features, and splice them into fusion modal representations for graphic and text retrieval.
Through modal fusion, the image and text features can be combined more effectively, the robustness and retrieval performance of graphic and text search is enhanced, the graphic and text content can be more comprehensively understood, and the retrieval decision process can be explained more clearly through attention conversion mechanisms.
Smart Images
Figure CN119396944A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image and text retrieval technology, and more specifically, to an image and text retrieval method and system based on big data. Background Art
[0002] Big data-based image and text retrieval technology integrates the latest advances in computer vision, natural language processing, and machine learning. By analyzing and processing large-scale image and text data, it helps users quickly find relevant content from massive information. This method is widely used in e-commerce, social media, digital libraries and other fields, bringing users a richer and more intuitive retrieval experience.
[0003] The core of image and text retrieval technology based on big data lies in the feature extraction and fusion of images and texts, which are usually processed using deep learning models (such as convolutional neural networks and Transformer architectures). After feature extraction, the system constructs a unified multimodal representation space so that images and texts can be compared and retrieved on the same dimension. Ultimately, the system provides users with the most relevant results by calculating the similarity between the query and the contents of each item in the data set. In the prior art, in image and text retrieval, the heterogeneity of image data and text data makes feature extraction and fusion complicated, and different data types and sources will produce huge feature differences, which increases the difficulty of image and text retrieval. Therefore, how to perform modal fusion of image modal information and text modal information to improve retrieval performance during image and text retrieval is a difficult problem faced by the industry. Summary of the invention
[0004] The present application provides a big data-based image and text retrieval method and system, which can perform modal fusion of image modal information and text modal information to improve the retrieval performance during image and text retrieval.
[0005] In a first aspect, the present application provides a method for image and text retrieval based on big data, comprising the following steps:
[0006] Obtain the graphic and text modal data to be retrieved;
[0007] Performing feature extraction on the image modality data and the text modality data in the image-text modality data respectively to obtain corresponding image feature representation and text feature representation, and determining the image-text modality fusion entropy between the image modality information and the text modality information through the image feature representation and the text feature representation;
[0008] Determine the text description array corresponding to the image feature representation according to the image-text modal fusion entropy, determine the image description array corresponding to the text feature representation according to the image-text modal fusion entropy, perform attention conversion according to the text description array and the image description array, and then obtain local image-text fusion features and global image-text fusion features in a two-dimensional modal space;
[0009] The local image-text fusion feature and the global image-text fusion feature are spliced to obtain a fusion modality representation between the image modality information and the text modality information, and image-text retrieval is performed based on the fusion modality representation.
[0010] In some embodiments, feature extraction is performed on the image modality data and the text modality data in the graphic and text modality data to obtain corresponding image feature representation and text feature representation, specifically including:
[0011] Converting the image modality data in the graphic modality data into an image modality vector;
[0012] Inputting the image modality vector into a convolutional neural network for feature extraction to obtain a corresponding image feature representation;
[0013] Converting text modality data in the graphic modality data into a text modality vector;
[0014] The text modality vector is input into a deep learning model for feature extraction to obtain a corresponding text feature representation.
[0015] In some embodiments, determining the text description array corresponding to the image feature representation according to the image-text modality fusion entropy specifically includes:
[0016] For each image feature vector in the image feature representation, obtaining all image-text modality fusion entropy values corresponding to the image feature vector;
[0017] Determine the text description corresponding to the image feature vector according to all image-text modality fusion entropy values corresponding to the image feature vector and the text feature representation, and then obtain the text description corresponding to each image feature vector;
[0018] The text description array corresponding to the image feature representation is determined through the text descriptions corresponding to all image feature vectors.
[0019] In some embodiments, determining the image description array corresponding to the text feature representation according to the image-text modality fusion entropy specifically includes:
[0020] For each text feature vector in the text feature representation, obtaining all image-text modality fusion entropy values corresponding to the text feature vector;
[0021] Determine the image description corresponding to the text feature vector according to all image-text modality fusion entropy values corresponding to the text feature vector and the image feature representation, and then obtain the image description corresponding to each text feature vector;
[0022] The image description array corresponding to the text feature representation is determined through the image descriptions corresponding to all text feature vectors.
[0023] In some embodiments, performing attention conversion according to the text description array and the image description array to obtain local image-text fusion features and global image-text fusion features in a two-dimensional modal space specifically includes:
[0024] Determine the local attention weight of each text description when the image and text are fused according to the text description array;
[0025] According to the local attention weights of all text descriptions when the image and text are fused, the text description array is converted into local image features when the text modality information is fused:
[0026] Determining a local attention weight of each image description when image and text are fused according to the image description array;
[0027] According to the local attention weights of all image descriptions when the image and text are fused, the image description array is converted into local text features when the image modality information is fused:
[0028] Constructing a local image-text fusion feature in a two-dimensional modal space through the local image feature and the local text feature;
[0029] Determine the global attention weight of each text description when the text and image are fused according to the text description array;
[0030] According to the global attention weight of all text descriptions when the image and text are fused, the text description array is converted into a global image feature when the text modality information is fused:
[0031] Determine the global attention weight of each image description when the image and text are fused according to the image description array;
[0032] According to the global attention weights of all image descriptions when the image and text are fused, the image description array is converted into a global text feature when the image modality information is fused:
[0033] A global image-text fusion feature in a two-dimensional modal space is constructed by using the global image feature and the global text feature.
[0034] In some embodiments, the local image-text fusion feature and the global image-text fusion feature are spliced to obtain a fusion modality representation between the image modality information and the text modality information, specifically including:
[0035] splicing the local image features in the local image-text fusion features and the global image features in the global image-text fusion features to obtain a final image representation;
[0036] The local text features in the local image-text fusion features and the global text features in the global image-text fusion features are concatenated to obtain a final text representation;
[0037] A fused modality representation between the image modality information and the text modality information is determined according to the final image representation and the final text representation.
[0038] In some embodiments, performing image and text retrieval based on the fused modality representation is to use the fused modality representation to construct an image and text index for image and text retrieval.
[0039] In a second aspect, the present application provides a big data-based image-text retrieval system, including an image-text fusion unit, wherein the image-text fusion unit includes:
[0040] An acquisition module, used to acquire the graphic and text modal data to be retrieved;
[0041] A processing module, used for performing feature extraction on the image modality data and the text modality data in the image-text modality data respectively, obtaining corresponding image feature representation and text feature representation, and determining the image-text modality fusion entropy between the image modality information and the text modality information through the image feature representation and the text feature representation;
[0042] The processing module is further used to determine the text description array corresponding to the image feature representation according to the image-text modal fusion entropy, determine the image description array corresponding to the text feature representation according to the image-text modal fusion entropy, perform attention conversion according to the text description array and the image description array, and then obtain local image-text fusion features and global image-text fusion features in a two-dimensional modal space;
[0043] The retrieval module is used to splice the local image-text fusion feature and the global image-text fusion feature to obtain a fusion modality representation between image modality information and text modality information, and perform image-text retrieval based on the fusion modality representation.
[0044] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned big data-based image and text retrieval method.
[0045] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions or codes are stored. When the instructions or codes are run on a computer, the computer implements the above-mentioned big data-based image and text retrieval method when executed.
[0046] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:
[0047] The present application provides a big data-based image-text retrieval method and system, which obtains image-text modal data to be retrieved; performs feature extraction on image modal data and text modal data in the image-text modal data respectively to obtain corresponding image feature representation and text feature representation, and determines the image-text modal fusion entropy between image modal information and text modal information through the image feature representation and the text feature representation; determines a text description array corresponding to the image feature representation according to the image-text modal fusion entropy, determines an image description array corresponding to the text feature representation according to the image-text modal fusion entropy, performs attention conversion according to the text description array and the image description array, and then obtains local image-text fusion features and global image-text fusion features in a two-dimensional modal space; splices the local image-text fusion features and the global image-text fusion features to obtain a fusion modal representation between the image modal information and the text modal information, and performs image-text retrieval based on the fusion modal representation.
[0048] It can be seen that in this application, image modal information and text modal information can be modally fused; wherein, based on the image-text modal fusion entropy, the correlation and mutual information between image features and text features can be measured, and the information of the two modalities can be more effectively combined, and the image-text modal fusion entropy takes into account the intrinsic connection between image and text, and enhances the robustness of image-text retrieval; then, according to the image description array and text description array determined by the modal fusion entropy, the multi-level semantic association between image and text can be deeply excavated, the local features focus on details, and the global features provide overall semantics, the combination of the two can fully understand the image and text content, and by paying attention to the conversion mechanism, the decision-making process of image-text retrieval can be explained more clearly; finally, the splicing of local features and global features increases the diversity of features, making the fused modal representation more extensive and comprehensive, and helping to capture information at different levels and dimensions. In summary, the technical solution adopted in this application can modally fuse image modal information and text modal information to improve the retrieval performance during image-text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0050] Figure 1 is an exemplary flow chart of a method for image and text retrieval based on big data according to some embodiments of the present application;
[0051] Figure 2 is an exemplary flow chart of determining image feature representation and text feature representation according to some embodiments of the present application;
[0052] Figure 3 is an exemplary flow chart of determining a fused modality representation according to some embodiments of the present application;
[0053] Figure 4 is a schematic diagram of exemplary hardware and / or software of a graphic-text fusion unit according to some embodiments of the present application;
[0054] Figure 5 It is a structural diagram of a computer device for implementing a big data-based image and text retrieval method as shown in some embodiments of the present application. DETAILED DESCRIPTION
[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0056] The embodiment of the present application provides a method and system for image-text retrieval based on big data, the core of which is to obtain image-text modal data to be retrieved; perform feature extraction on the image modal data and text modal data in the image-text modal data respectively to obtain corresponding image feature representation and text feature representation, determine the image-text modal fusion entropy between the image modal information and the text modal information through the image feature representation and the text feature representation; determine the text description array corresponding to the image feature representation according to the image-text modal fusion entropy, determine the image description array corresponding to the text feature representation according to the image-text modal fusion entropy, perform attention conversion according to the text description array and the image description array, and then obtain local image-text fusion features and global image-text fusion features in a two-dimensional modal space; splice the local image-text fusion features and the global image-text fusion features to obtain the fusion modal representation between the image modal information and the text modal information, and perform image-text retrieval based on the fusion modal representation. The above scheme can be used to perform modal fusion of image modal information and text modal information to improve the retrieval performance during image-text retrieval.
[0057] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of a method for image and text retrieval based on big data according to some embodiments of the present application. The method 100 for image and text retrieval based on big data mainly includes the following steps:
[0058] In step 101, the graphic and text modal data to be retrieved is obtained.
[0059] In specific implementation, a web crawler tool can be used to automatically crawl data from the Internet. This method can obtain a large number of images and associated texts (i.e., graphic and text modal data); it should be noted that in this application, graphic and text modal data includes image modal data and text modal data, image modal data is represented as images, and text modal data is represented as text. Web crawlers need to pay attention to complying with the website's robot exclusion agreement and relevant laws and regulations to avoid infringement of copyright and privacy.
[0060] In step 102, feature extraction is performed on the image modal data and the text modal data in the graphic-text modal data respectively to obtain corresponding image feature representation and text feature representation, and the graphic-text modal fusion entropy between the image modal information and the text modal information is determined through the image feature representation and the text feature representation.
[0061] Preferably, in some embodiments, reference Figure 2As shown in FIG. 1 , this figure is an exemplary flow chart for determining image feature representation and text feature representation in some embodiments of the present application. In this embodiment, feature extraction is performed on the image modality data and the text modality data in the image-text modality data respectively, and the corresponding image feature representation and text feature representation can be obtained by the following steps:
[0062] First, in step 1021, the image modality data in the graphic modality data is converted into an image modality vector;
[0063] Then, in step 1022, the image modality vector is input into a convolutional neural network for feature extraction to obtain a corresponding image feature representation;
[0064] Secondly, in step 1023, the text modality data in the graphic modality data is converted into a text modality vector;
[0065] Finally, in step 1024, the text modality vector is input into the deep learning model for feature extraction to obtain the corresponding text feature representation.
[0066] It should be noted that, in the present application, the image feature representation includes an image feature vector corresponding to each image block; the text feature representation includes a text feature vector corresponding to each text word.
[0067] In the specific implementation, first, for the image modality data in the graphic and text modality data, the pixels of the image modality data are adjusted to 224*224, the image is divided into image blocks of size 16*16 and quantity 196, and then the position information encoding is added to all the image blocks and the image blocks are sequentially expanded and converted into vectors, namely, image modality vectors; then, the image modality vectors can be input into a convolutional neural network for feature extraction to obtain the corresponding image feature representation; secondly, the text modality data in the graphic and text modality data can be input into a word2vec model to be converted into a word vector, namely, a text modality vector; finally, the text modality vector can be input into a deep learning model for feature extraction to obtain the corresponding text feature representation.
[0068] In some embodiments, the image-text modality fusion entropy between the image modality information and the text modality information is determined by using the image feature representation and the text feature representation in the following manner, namely:
[0069] For each image feature vector in the image feature representation, determine all image-text modality fusion entropy values corresponding to the image feature vector according to the image feature vector and all text feature vectors in the text feature representation, and then obtain all image-text modality fusion entropy values corresponding to each image feature vector in the image feature representation;
[0070] The image-text modality fusion entropy between the image modality information and the text modality information is determined by taking all image-text modality fusion entropy values corresponding to each image feature vector in the image feature representation.
[0071] It should be noted that in the present application, the image-text modal fusion entropy represents the degree of information overlap between the image modal information and the text modal information. The larger the image-text modal fusion entropy, the higher the degree of information overlap, and the higher the similarity between the image feature representation and the text feature representation. After obtaining all values of the image-text modal fusion entropy, the image-text modal fusion entropy between the image modal information and the text modal information can be obtained.
[0072] In the specific implementation, first, for each image feature vector in the image feature representation, the transpose of the image feature vector can be multiplied by all the text feature vectors in the text feature representation, and the result can be used as the image-text modal fusion entropy value corresponding to the image feature vector, thereby obtaining all the image-text modal fusion entropy values corresponding to the image feature vector. In the above manner, all the image-text modal fusion entropy values corresponding to each image feature vector in the image feature representation can be obtained; then, all the image-text modal fusion entropy values corresponding to each image feature vector in the image feature representation can be used as the value interval of the image-text modal fusion entropy between the image modal information and the text modal information, so as to obtain the image-text modal fusion entropy between the image modal information and the text modal information. It should be noted that each image feature vector corresponds to multiple image-text modal fusion entropy values, and each text feature vector also corresponds to multiple image-text modal fusion entropy values.
[0073] It should be noted that the image-text modal fusion entropy can measure the correlation and mutual information between image features and text features, and can more effectively combine the information of the two modalities to generate a richer and more complete representation. In addition, the image-text modal fusion entropy takes into account the intrinsic connection between images and texts, which reduces the model's dependence on single modal information and enhances the robustness of image-text retrieval.
[0074] In step 103, the text description array corresponding to the image feature representation is determined according to the image-text modal fusion entropy, the image description array corresponding to the text feature representation is determined according to the image-text modal fusion entropy, and attention conversion is performed based on the text description array and the image description array, thereby obtaining local image-text fusion features and global image-text fusion features in a two-dimensional modal space.
[0075] Optionally, in some embodiments, determining the text description array corresponding to the image feature representation according to the image-text modality fusion entropy may specifically be performed in the following manner, namely:
[0076] For each image feature vector in the image feature representation, obtaining all image-text modality fusion entropy values corresponding to the image feature vector;
[0077] Determine the text description corresponding to the image feature vector according to all image-text modality fusion entropy values corresponding to the image feature vector and the text feature representation, and then obtain the text description corresponding to each image feature vector;
[0078] The text description array corresponding to the image feature representation is determined through the text descriptions corresponding to all image feature vectors.
[0079] In specific implementation, first, for each image feature vector in the image feature representation, all image-text modality fusion entropy values corresponding to the image feature vector can be obtained by the method in step 102; then, the text description corresponding to the image feature vector can be determined according to all image-text modality fusion entropy values corresponding to the image feature vector and the text feature representation, for example, the text description α′ corresponding to the i-th image feature vector in the image feature representation i It can be determined according to the following formula:
[0080]
[0081] Among them, β j represents the jth text feature vector in the text feature representation, λ ij represents the j-th image-text modality fusion entropy value corresponding to the ith image feature vector, m represents the total number of text feature vectors in the text feature representation, and the text description is the text description feature of the image block corresponding to the image feature vector, that is, the image block corresponding to the image feature vector is described by text features. The text description corresponding to each image feature vector can be obtained in the above manner; finally, the matrix composed of the text descriptions corresponding to all image feature vectors can be used as the text description array corresponding to the image feature representation.
[0082] Optionally, in some embodiments, determining the image description array corresponding to the text feature representation according to the image-text modality fusion entropy may specifically be performed in the following manner, namely:
[0083] For each text feature vector in the text feature representation, obtaining all image-text modality fusion entropy values corresponding to the text feature vector;
[0084] Determine the image description corresponding to the text feature vector according to all image-text modality fusion entropy values corresponding to the text feature vector and the image feature representation, and then obtain the image description corresponding to each text feature vector;
[0085] The image description array corresponding to the text feature representation is determined through the image descriptions corresponding to all text feature vectors.
[0086] In the specific implementation, first, for each text feature vector in the text feature representation, all the image-text modality fusion entropy values corresponding to the text feature vector can be obtained by the method in step 102; then, the image description corresponding to the text feature vector can be determined according to all the image-text modality fusion entropy values corresponding to the text feature vector and the image feature representation, for example, the image description β′ corresponding to the j-th text feature vector in the text feature representation j It can be determined according to the following formula:
[0087]
[0088] Among them, α i represents the i-th image feature vector in the image feature representation, λ ij represents the j-th image-text modality fusion entropy value corresponding to the ith image feature vector, n represents the total number of image feature vectors in the image feature representation, and the image description is the image description feature of the text word corresponding to the text feature vector, that is, the text word corresponding to the text feature vector is described by the image feature. The image description corresponding to each text feature vector can be obtained in the above manner; finally, the matrix composed of the image descriptions corresponding to all text feature vectors can be used as the image description array corresponding to the text feature representation.
[0089] Preferably, in some embodiments, the attention conversion is performed according to the text description array and the image description array, and then the local image-text fusion feature and the global image-text fusion feature in the two-dimensional modal space are obtained in the following manner, namely:
[0090] Determine the local attention weight of each text description when the image and text are fused according to the text description array;
[0091] According to the local attention weights of all text descriptions when the image and text are fused, the text description array is converted into local image features when the text modality information is fused:
[0092] Determining a local attention weight of each image description when image and text are fused according to the image description array;
[0093] According to the local attention weights of all image descriptions when the image and text are fused, the image description array is converted into local text features when the image modality information is fused:
[0094] Constructing a local image-text fusion feature in a two-dimensional modal space through the local image feature and the local text feature;
[0095] Determine the global attention weight of each text description when the text and image are fused according to the text description array;
[0096] According to the global attention weight of all text descriptions when the image and text are fused, the text description array is converted into a global image feature when the text modality information is fused:
[0097] Determine the global attention weight of each image description when the image and text are fused according to the image description array;
[0098] According to the global attention weights of all image descriptions when the image and text are fused, the image description array is converted into a global text feature when the image modality information is fused:
[0099] A global image-text fusion feature in a two-dimensional modal space is constructed by using the global image feature and the global text feature.
[0100] In specific implementation, first, the mean of all text descriptions in the text description array can be calculated. For each text description, the ratio of the text description to the mean of all text descriptions and the product of the local image features are used as the local attention weight of the text description when the text and image are fused. The local attention weight of each text description when the text and image are fused can be obtained in the above manner; then, the local attention weight of each text description when the text and image are fused can be multiplied by the corresponding text description in the text description array, and the sum of all results can be used as the local image feature when fusing the text modal information: secondly, the local attention weight of each image description in the image description array when the text and image are fused can be calculated in the above manner; then, each image description can be multiplied by the local attention weight of each text description when the text and image are fused. The local attention weight during image and text fusion is multiplied by the corresponding image description in the image description array, and the sum of all results is used as the local text feature when fusing the image modal information: Finally, the local image and text fusion features in the two-dimensional modal space can be constructed by the local image features and the local text features, that is, the feature vector composed of the local image features and the local text features is used as the local image and text fusion features in the two-dimensional modal space; It should be noted that in the present application, the process of constructing the global image and text fusion features in the two-dimensional modal space is the same as the process of constructing the local image and text fusion features in the two-dimensional modal space, and can be derived through the process of constructing the local image and text fusion features in the two-dimensional modal space. It only needs to replace the different parameters, which will not be repeated here.
[0101] It should be noted that the image description array and text description array determined by the modal fusion entropy can deeply explore the multi-level semantic associations between images and texts. Local features focus on details, while global features provide overall semantics. The combination of the two can fully understand the content of images and texts, and by paying attention to the conversion mechanism, the decision-making process of image and text retrieval can be more clearly explained, and it can be understood which image and text features play a key role in the final fusion features and retrieval results.
[0102] In step 104, the local image-text fusion feature and the global image-text fusion feature are spliced to obtain a fusion modality representation between image modality information and text modality information, and image-text retrieval is performed based on the fusion modality representation.
[0103] Preferably, in some embodiments, reference Figure 3 As shown in FIG. 1 , this figure is an exemplary flow chart for determining a fusion modality representation in some embodiments of the present application. In this embodiment, the local image-text fusion feature and the global image-text fusion feature are spliced to obtain a fusion modality representation between the image modality information and the text modality information, which can be implemented by the following steps:
[0104] First, in step 1041, the local image features in the local image-text fusion features and the global image features in the global image-text fusion features are spliced to obtain a final image representation;
[0105] Then, in step 1042, the local text features in the local image-text fusion features and the global text features in the global image-text fusion features are concatenated to obtain a final text representation;
[0106] Finally, in step 1043, a fused modality representation between the image modality information and the text modality information is determined based on the final image representation and the final text representation.
[0107] It should be noted that in the present application, the final image representation is obtained by feature splicing to obtain image feature representation; the final text representation is obtained by feature splicing to obtain text feature representation; the fused modal representation is a feature representation that fuses image modal information and text modal information. The fused modal representation takes into account the mutual influence between modalities, can fuse the features of different modalities and explore the intrinsic connections between different modalities, and more comprehensively represent the characteristics of graphic and text data.
[0108] In the specific implementation, first, the existing feature stitching technology (horizontal stitching) can be used to stitch the image features (local image features and global image features) and text features (local text features and global text features) respectively, which will not be repeated here; then, the fused modal representation between the image modal information and the text modal information can be determined according to the final image representation and the final text representation, that is, the final image representation and the final text representation can be spliced together to obtain a feature representation, and the feature representation can be used as the fused modal representation between the image modal information and the text modal information.
[0109] In some embodiments, performing image and text retrieval based on the fused modal representation is to use the fused modal representation to construct an image and text index for image and text retrieval; in specific implementation, first, an index is constructed based on the fused modal representation, and an efficient data structure (such as an inverted index, LSH, etc.) can be used to store the representation and the corresponding image and text data; then, the user's query (which can be text, image, or a mixture of image and text) is converted into a fused modal representation, and the similarity between the fused modal representation of the query and the stored representation is calculated in the index (such as cosine similarity, Euclidean distance, etc.); finally, the retrieval results are sorted according to the similarity score, and the most relevant image and text data is returned to complete the image and text retrieval.
[0110] It can be seen that in this application, image modal information and text modal information can be modally fused; wherein, based on the image-text modal fusion entropy, the correlation and mutual information between image features and text features can be measured, and the information of the two modalities can be more effectively combined, and the image-text modal fusion entropy takes into account the intrinsic connection between image and text, and enhances the robustness of image-text retrieval; then, according to the image description array and text description array determined by the modal fusion entropy, the multi-level semantic association between image and text can be deeply excavated, the local features focus on details, and the global features provide overall semantics, the combination of the two can fully understand the image and text content, and by paying attention to the conversion mechanism, the decision-making process of image-text retrieval can be explained more clearly; finally, the splicing of local features and global features increases the diversity of features, making the fused modal representation more extensive and comprehensive, and helping to capture information at different levels and dimensions. In summary, the technical solution adopted in this application can modally fuse image modal information and text modal information to improve the retrieval performance during image-text retrieval.
[0111] In addition, in another aspect of the present application, in some embodiments, the present application provides a big data-based image-text retrieval system, the big data-based image-text retrieval system includes an image-text fusion unit, reference Figure 4 , which is a schematic diagram of exemplary hardware and / or software of a text-image fusion unit according to some embodiments of the present application, the text-image fusion unit 400 includes: an acquisition module 401, a processing module 402 and a retrieval module 403, which are described as follows:
[0112] Acquisition module 401, in this application, acquisition module 401 is mainly used to acquire graphic and text modal data to be retrieved;
[0113] Processing module 402, in the present application, the processing module 402 is used to perform feature extraction on the image modality data and the text modality data in the image-text modality data, respectively, to obtain corresponding image feature representation and text feature representation, and to determine the image-text modality fusion entropy between the image modality information and the text modality information through the image feature representation and the text feature representation;
[0114] It should be noted that the processing module 402 in the present application is also used to determine the text description array corresponding to the image feature representation according to the image-text modal fusion entropy, determine the image description array corresponding to the text feature representation according to the image-text modal fusion entropy, perform attention conversion according to the text description array and the image description array, and then obtain the local image-text fusion feature and the global image-text fusion feature in the two-dimensional modal space;
[0115] Retrieval module 403, in the present application, the retrieval module 403 is mainly used to splice the local image-text fusion features and the global image-text fusion features to obtain a fusion modal representation between image modal information and text modal information, and perform image-text retrieval based on the fusion modal representation.
[0116] The above describes in detail the examples of the image and text retrieval method and system based on big data provided by the embodiments of the present application. It can be understood that in order to realize the above functions, the corresponding device includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.
[0117] In some embodiments, the present application also provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the above-mentioned big data-based image and text retrieval method.
[0118] In some embodiments, reference Figure 5 , the dotted line in the figure indicates that the unit or the module is optional. The figure is a schematic diagram of the structure of a computer device for a method for image and text retrieval based on big data provided in an embodiment of the present application. The method for image and text retrieval based on big data described in the above embodiment can be performed by Figure 5 The computer device 500 shown in the figure is implemented, and the computer device 500 includes at least one processor 501, a memory 502 and at least one communication unit 505. The computer device 500 can be a terminal device, a server or a chip.
[0119] The processor 501 may be a general-purpose processor or a special-purpose processor. For example, the processor 501 may be a central processing unit (CPU), which may be used to control the computer device 500, execute software programs, and process data of the software programs. The computer device 500 may also include a communication unit 505 to implement signal input (reception) and output (transmission).
[0120] For example, the computer device 500 may be a chip, the communication unit 505 may be an input and / or output circuit of the chip, or the communication unit 505 may be a communication interface of the chip, and the chip may be a component of a terminal device, a network device, or other devices.
[0121] For another example, the computer device 500 may be a terminal device or a server, and the communication unit 505 may be a transceiver of the terminal device or the server, or the communication unit 505 may be a transceiver circuit of the terminal device or the server.
[0122] The computer device 500 may include one or more memories 502, on which a program 504 is stored. The program 504 can be executed by the processor 501 to generate instructions 503, so that the processor 501 performs the method described in the above method embodiment according to the instructions 503. Optionally, data (such as a target audit model) can also be stored in the memory 502. Optionally, the processor 501 can also read the data stored in the memory 502, and the data can be stored at the same storage address as the program 504, or the data can be stored at a different storage address from the program 504.
[0123] The processor 501 and the memory 502 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.
[0124] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or software-based instructions in the processor 501. The processor 501 can be a central processing unit, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, such as discrete gates, transistor logic devices or discrete hardware components.
[0125] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0126] For example, in some embodiments, the present application also provides a computer-readable storage medium, in which instructions or codes are stored. When the instructions or codes are run on a computer, the computer implements the above-mentioned big data-based image and text retrieval method when executed.
[0127] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0128] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for image and text retrieval based on big data, characterized in that: The steps include: Obtain the graphic and text modal data to be retrieved; Performing feature extraction on the image modality data and the text modality data in the image-text modality data respectively to obtain corresponding image feature representation and text feature representation, and determining the image-text modality fusion entropy between the image modality information and the text modality information through the image feature representation and the text feature representation; Determine the text description array corresponding to the image feature representation according to the image-text modal fusion entropy, determine the image description array corresponding to the text feature representation according to the image-text modal fusion entropy, perform attention conversion according to the text description array and the image description array, and then obtain local image-text fusion features and global image-text fusion features in a two-dimensional modal space; The local image-text fusion feature and the global image-text fusion feature are spliced to obtain a fusion modality representation between the image modality information and the text modality information, and image-text retrieval is performed based on the fusion modality representation.
2. The method according to claim 1, characterized in that Performing feature extraction on the image modality data and the text modality data in the graphic and text modality data respectively to obtain corresponding image feature representation and text feature representation specifically includes: Converting the image modality data in the graphic modality data into an image modality vector; Inputting the image modality vector into a convolutional neural network for feature extraction to obtain a corresponding image feature representation; Converting text modality data in the graphic modality data into a text modality vector; The text modality vector is input into a deep learning model for feature extraction to obtain a corresponding text feature representation.
3. The method according to claim 1, characterized in that Determining the text description array corresponding to the image feature representation according to the image-text modality fusion entropy specifically includes: For each image feature vector in the image feature representation, obtaining all image-text modality fusion entropy values corresponding to the image feature vector; Determine the text description corresponding to the image feature vector according to all image-text modality fusion entropy values corresponding to the image feature vector and the text feature representation, and then obtain the text description corresponding to each image feature vector; The text description array corresponding to the image feature representation is determined through the text descriptions corresponding to all image feature vectors.
4. The method according to claim 1, characterized in that Determining the image description array corresponding to the text feature representation according to the image-text modality fusion entropy specifically includes: For each text feature vector in the text feature representation, obtaining all image-text modality fusion entropy values corresponding to the text feature vector; Determine the image description corresponding to the text feature vector according to all image-text modality fusion entropy values corresponding to the text feature vector and the image feature representation, and then obtain the image description corresponding to each text feature vector; The image description array corresponding to the text feature representation is determined through the image descriptions corresponding to all text feature vectors.
5. The method according to claim 1, characterized in that Attention conversion is performed according to the text description array and the image description array, thereby obtaining local image-text fusion features and global image-text fusion features in a two-dimensional modal space, specifically including: Determine the local attention weight of each text description when the image and text are fused according to the text description array; According to the local attention weights of all text descriptions when the image and text are fused, the text description array is converted into local image features when the text modality information is fused: Determining a local attention weight of each image description when image and text are fused according to the image description array; According to the local attention weights of all image descriptions when the image and text are fused, the image description array is converted into local text features when the image modality information is fused: Constructing a local image-text fusion feature in a two-dimensional modal space through the local image feature and the local text feature; Determine the global attention weight of each text description when the text and image are fused according to the text description array; According to the global attention weight of all text descriptions when the image and text are fused, the text description array is converted into a global image feature when the text modality information is fused: Determine the global attention weight of each image description when the image and text are fused according to the image description array; According to the global attention weights of all image descriptions when the image and text are fused, the image description array is converted into a global text feature when the image modality information is fused: A global image-text fusion feature in a two-dimensional modal space is constructed by using the global image feature and the global text feature.
6. The method according to claim 1, characterized in that The local image-text fusion feature and the global image-text fusion feature are spliced to obtain a fusion modality representation between the image modality information and the text modality information, which specifically includes: splicing the local image features in the local image-text fusion features and the global image features in the global image-text fusion features to obtain a final image representation; The local text features in the local image-text fusion features and the global text features in the global image-text fusion features are concatenated to obtain a final text representation; A fused modality representation between the image modality information and the text modality information is determined according to the final image representation and the final text representation.
7. The method according to claim 1, characterized in that The image-text retrieval based on the fused modality representation is to use the fused modality representation to construct an image-text index for image-text retrieval.
8. A big data-based image-text retrieval system, the big data-based image-text retrieval system comprising an image-text fusion unit, characterized in that: The image-text fusion unit comprises: An acquisition module, used to acquire the graphic and text modal data to be retrieved; A processing module, used for performing feature extraction on the image modality data and the text modality data in the image-text modality data respectively, obtaining corresponding image feature representation and text feature representation, and determining the image-text modality fusion entropy between the image modality information and the text modality information through the image feature representation and the text feature representation; The processing module is further used to determine the text description array corresponding to the image feature representation according to the image-text modal fusion entropy, determine the image description array corresponding to the text feature representation according to the image-text modal fusion entropy, perform attention conversion according to the text description array and the image description array, and then obtain local image-text fusion features and global image-text fusion features in a two-dimensional modal space; The retrieval module is used to splice the local image-text fusion feature and the global image-text fusion feature to obtain a fusion modality representation between image modality information and text modality information, and perform image-text retrieval based on the fusion modality representation.
9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the computer device executes the image and text retrieval method based on big data as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions or codes, and when the instructions or codes are executed on a computer, the computer implements the image and text retrieval method based on big data as described in any one of claims 1 to 7.