Image-text matching method and device, electronic equipment, storage medium and computer program product

Abstract

The invention provides an image-text matching method and device, electronic equipment, a storage medium and a computer program product. The method comprises the steps of performing semantic analysis on a text, and determining keywords in the text and weights of the keywords on semantic influences; performing image-text matching on each keyword, and determining the similarity between each keyword and each candidate picture in the candidate picture set; based on the similarity between each keyword and each candidate picture in the candidate picture set and the weights of the keywords, the fusion similarity between each candidate picture and all the keywords is determined; and based on the fusion similarity of each candidate picture, screening out a target picture matched with the text from the candidate picture set. Through the image-text matching method and device, the image-text matching accuracy can be improved.

Description

Technical Field

[0001] This application relates to data processing technology, and more particularly to a method, apparatus, electronic device, storage medium, and computer program product for image and text matching. Background Technology

[0002] With the development of information technology, the demand for image-text matching is increasing. For example, users can quickly find images in their local image library that match the text content by entering search terms, or they can use the text content of an article to find matching images online and insert them into the article. However, the accuracy of image-text matching still needs to be improved. Summary of the Invention

[0003] This application provides a text-image matching method, apparatus, electronic device, storage medium, and computer program product that can save computing power and reduce power consumption while minimizing the loss of prediction accuracy.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides an image-text matching method, including:

[0006] Perform semantic analysis on the text to determine the keywords in the text and the weight of each keyword's semantic influence;

[0007] Perform image-text matching for each keyword and determine the similarity between each keyword and each candidate image in the candidate image set;

[0008] Based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword, the fusion similarity between each candidate image and all keywords is determined.

[0009] Based on the fusion similarity of each candidate image, target images that match the text are selected from the candidate image set.

[0010] In some embodiments, determining the fusion similarity between each candidate image and all keywords based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword, includes:

[0011] A non-linear mapping is performed on the similarity between each candidate image and each keyword to obtain a similarity mapping value between each candidate image and each keyword; wherein, if the similarity before mapping is greater than or equal to a first preset similarity threshold, the similarity mapping value after mapping is greater than the similarity; if the similarity is less than the first preset similarity threshold, the similarity mapping value is less than the similarity; the similarity mapping value is positively correlated with the similarity.

[0012] The similarity mapping values ​​between candidate images and keywords are weighted using the weight of keywords to obtain the weighted similarity between each candidate image and each keyword;

[0013] For each candidate image, the weighted similarity with each keyword is summed to obtain the fusion similarity between the candidate image and all keywords.

[0014] In some embodiments, the step of filtering target images that match the text from the candidate image set based on the fusion similarity of each candidate image includes:

[0015] Cluster analysis is performed on the candidate images based on their content to determine the clusters.

[0016] Based on the fusion similarity of candidate images in each cluster, the fusion similarity statistics corresponding to the cluster are determined.

[0017] Based on the fusion similarity statistics of each cluster, target images that match the text are selected.

[0018] In some embodiments, the step of performing cluster analysis on candidate images based on the image content of each candidate image to determine clusters includes:

[0019] The number of clusters K is preset. Clustering is performed based on the image content of each candidate image, and the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters is determined.

[0020] In response to the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters, the decrease in the total intra-class variance of the K-1 clusters relative to the total intra-class variance of the K-2 clusters becomes smaller, and the difference between the two is less than a preset difference threshold, the K clusters are determined as the clusters.

[0021] In some embodiments, determining the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters includes:

[0022] The difference between the intra-class variance when all target candidate images are clustered into one class and the total intra-class variance when all target candidate images are clustered into a preset maximum number of classes is used as the denominator. The difference between the total intra-class variance of K-1 clusters and the total intra-class variance of K clusters is used as the numerator. The decrease magnitude is determined based on the ratio of the numerator and the denominator.

[0023] In some embodiments, the step of performing cluster analysis on candidate images based on the image content of each candidate image to determine clusters includes:

[0024] Cluster analysis is performed on candidate images based on their content, where the fusion similarity is greater than a preset second similarity threshold, to determine the clusters.

[0025] In some embodiments, the step of filtering out target images that match the text based on the fusion similarity statistics of each cluster includes:

[0026] The image from the cluster with the highest fusion similarity score among all clusters is selected as the target image for matching the text.

[0027] In some embodiments, determining the fusion similarity between each candidate image and all keywords based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword, includes:

[0028] From the candidate image set, candidate images whose similarity to the target keyword is greater than a third preset similarity threshold are selected; wherein, the target keyword is a keyword with a weight greater than a preset weight threshold;

[0029] For each selected candidate image, the fusion similarity between each candidate image and all keywords is determined based on the similarity between the candidate image and each keyword, as well as the weight of the keyword.

[0030] This application provides an image-text matching device, including:

[0031] The semantic analysis module is configured to perform semantic analysis on the text to determine the keywords in the text and the weight of each keyword's semantic influence.

[0032] The image-text matching module is configured to perform image-text matching for each keyword and determine the similarity between each keyword and each candidate image in the candidate image set.

[0033] The fusion similarity determination module is configured to determine the fusion similarity between each candidate image and all keywords based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword.

[0034] The filtering module is configured to filter out target images that match the text from the candidate image set based on the fusion similarity of each candidate image.

[0035] In some embodiments, the fusion similarity determination module is further configured to perform a non-linear mapping on the similarity between each candidate image and each keyword to obtain a similarity mapping value between each candidate image and each keyword; wherein, if the similarity before mapping is greater than or equal to a first preset similarity threshold, the similarity mapping value after mapping is greater than the similarity; if the similarity is less than the first preset similarity threshold, the similarity mapping value is less than the similarity; the similarity mapping value is positively correlated with the similarity; the similarity mapping values ​​between candidate images and keywords are weighted using the weights of keywords to obtain a weighted similarity between each candidate image and each keyword; for each candidate image, the weighted similarity with each keyword is added to obtain the fusion similarity between the candidate image and all keywords.

[0036] In some embodiments, the filtering module is further configured to perform cluster analysis on the candidate images based on the image content of each candidate image to determine clusters; determine the fusion similarity statistics of the clusters based on the fusion similarity of the candidate images in each cluster; and filter out target images that match the text based on the fusion similarity statistics of each cluster.

[0037] In some embodiments, the filtering module is further configured to preset the number of clusters K, perform clustering based on the image content of each candidate image, and determine the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters; in response to the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters being smaller than the decrease in the total intra-class variance of the K-2 clusters relative to the total intra-class variance of the K-1 clusters, and the difference between the two being less than a preset difference threshold, the K clusters are determined as the clusters.

[0038] In some embodiments, the filtering module is further configured to use the difference between the intra-class variance when all target candidate images are clustered into one class and the total intra-class variance when all target candidate images are clustered into a preset maximum number of classes as the denominator, and the difference between the total intra-class variance of K-1 clusters and the total intra-class variance of K clusters as the numerator, and determine the decrease magnitude based on the ratio of the numerator and the denominator.

[0039] In some embodiments, the filtering module is further configured to perform cluster analysis on the candidate images based on the image content of each candidate image whose fusion similarity is greater than a preset second similarity threshold, and determine the cluster.

[0040] In some embodiments, the filtering module is further configured to select the image from the cluster with the highest fusion similarity statistical value among all clusters as the target image to match the text.

[0041] In some embodiments, the fusion similarity determination module is further configured to select candidate images from the candidate image set whose similarity to the target keyword is greater than a third preset similarity threshold; wherein the target keyword is a keyword with a weight greater than a preset weight threshold; and for each selected candidate image, the fusion similarity between each candidate image and all keywords is determined based on the similarity between the candidate image and each keyword and the weight of the keyword.

[0042] This application provides an electronic device, including:

[0043] processor;

[0044] Memory used to store computer programs or instructions;

[0045] The processor executes the computer program or instructions to implement the method provided in the embodiments of this application.

[0046] This application provides a computer-readable storage medium storing a computer program or instructions thereon, which, when executed by a processor, implements the method provided in this application.

[0047] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method provided in this application.

[0048] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0049] In this application, the electronic device first performs semantic analysis on the text. On the one hand, after identifying keywords, image-text matching is performed on each keyword, ensuring that every core keyword (semantic meaning) in the text is considered. Compared to directly performing image-text matching on the entire text, this reduces the likelihood of some core words being weakened during the overall text matching process. On the other hand, based on semantic analysis, weights are assigned to each keyword. After image-text matching on each keyword, the fusion similarity between each candidate image and all keywords is determined based on the weights of each keyword. Matching target images are then selected based on this fusion similarity. Compared to directly performing word segmentation matching on the text without considering weights, this application, while ensuring comprehensive matching (covering all keywords), highlights the dominant role of core semantic meaning through weights, thereby helping to select target images that better match the text from the candidate image set. The keyword-by-keyword matching in this embodiment, with weighted fusion similarity used to guide the selection process, improves the accuracy of the image-text matching results.

[0050] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0052] Figure 1 This is a schematic diagram of the architecture of a text-image matching system provided in an embodiment of this application;

[0053] Figure 2 This is a schematic diagram of the terminal structure provided in the embodiments of this application;

[0054] Figure 3 This is a schematic flowchart of the image-text matching method provided in the embodiments of this application;

[0055] Figure 4 This is a flowchart illustrating another image-text matching method provided in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0058] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0059] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0060] In the field of cross-modal image-text retrieval, the CLIP (Contrastive Language-Image Pre-training) model, a groundbreaking technology proposed by OpenAI, constructs a shared semantic space for images and text through a contrastive learning mechanism, providing a fundamental framework for cross-modal retrieval. This model employs a dual-encoder architecture (visual encoder and text encoder), utilizing large-scale image-text pair pre-training to achieve feature alignment between modalities. Compared to traditional linear retrieval methods based on a single text input, CLIP supports end-to-end cross-modal similarity calculation.

[0061] Image-text retrieval methods in related technologies typically use models such as CLIP to directly calculate the similarity between text and images. However, when the input is a complex long sentence, CLIP struggles to adaptively identify key semantic components and their differences in importance. For example, for the query "a brown puppy wearing a red hat running in the grass," traditional methods may only match images of "puppy," ignoring the influence of descriptive features (such as "red hat"), resulting in low matching accuracy.

[0062] This application provides a method, apparatus, electronic device, and computer-readable storage medium for image-text matching, which can improve the accuracy of image-text matching.

[0063] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of user terminal devices or edge devices, such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as servers. The following will describe exemplary applications when the electronic device is implemented as a server.

[0064] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the image and text matching system 100 provided in the embodiments of this application. In order to implement the image and text matching method of this application, the terminal (terminal 200-1 and terminal 200-2 are shown as examples) connects to the server 400 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0065] In some possible implementations, user A can trigger image-text matching request A through terminal 200-1, and user B can trigger image-text matching request B through terminal 200-2. Image-text matching A and image-text matching B are uploaded to server 400 via network 300. Image-text matching A can be triggered by the user entering text in the image search application of 210-1, thereby triggering server 400 to search for images corresponding to the text based on the image-text matching method of this application embodiment. Request B can be triggered by the user selecting a piece of text in the text creation application of 210-2 and indicating that an image should be provided for that text, thereby triggering server 400 to match images with the selected text based on the image-text matching method of this application embodiment. In some embodiments, server 400 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms. Terminal 200 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminals and servers can be connected directly or indirectly via wired or wireless communication methods, and no restrictions are imposed in this embodiment of the invention.

[0066] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the terminal 200 provided in the embodiments of this application. It should be noted that... Figure 2 The terminal 200 shown can be Figure 1 Either of the two terminals shown can be another terminal, and this application does not limit this. Figure 2 The terminal 200 shown includes at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in the terminal 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0067] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0068] User interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0069] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0070] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0071] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0072] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0073] The network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 including: Bluetooth, Wi-Fi, and Universal Serial Bus (USB), etc.

[0074] Presentation module 253 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 231 associated with user interface 230 (e.g., a display screen, a speaker, etc.).

[0075] The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0076] In some embodiments, the image-text matching method of this application can also be executed by a terminal, and the apparatus provided in the embodiments of this application can be implemented in software. Figure 2 A text-image matching device 255 stored in memory 250 is shown. This device can be software in the form of programs or plug-ins, and includes the following software modules: a semantic analysis module 2551, a text-image matching module 2552, a similarity determination module 2553, and a filtering module 2554. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0077] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0078] In some embodiments, the terminal or server can implement the methods provided in the embodiments of this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a local application (APP), that is, a program that needs to be installed in the operating system to run, such as a social APP; it can also be a mini-program, that is, a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0079] The following will describe the image-text matching method provided in this application embodiment by referring to exemplary applications and implementations of the electronic devices provided in the embodiments of this application.

[0080] See Figure 3 , Figure 3 This is a flowchart illustrating the image-text matching method provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained.

[0081] S301. Perform semantic analysis on the text to determine the keywords in the text and the weight of each keyword's semantic influence;

[0082] S302. Perform image-text matching for each keyword and determine the similarity between each keyword and each candidate image in the candidate image set.

[0083] S303. Based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword, determine the fusion similarity between each candidate image and all keywords.

[0084] S304. Based on the fusion similarity of each candidate image, select the target image that matches the text from the candidate image set.

[0085] The image-text matching method in this application can be applied to the aforementioned servers, user terminals, or edge devices, collectively referred to herein as electronic devices.

[0086] In step S301, the electronic device performs semantic analysis on the text, for example, based on a Large Language Model (LLM), to determine the keywords in the text and the weights of each keyword. Keywords can be some or all of the words in the text; keywords are crucial to the semantic expression of the text, and the keyword weights characterize the degree of influence of the keywords on the semantics. For example, words with high frequency of occurrence or words with special parts of speech may have higher weights; however, this embodiment does not impose such limitations.

[0087] In some embodiments, the large language model labels the keywords and their weights in each text sample during training. When using the large language model, inputting the text into the large language model will yield the keywords in the text and the weights of each keyword.

[0088] In other embodiments, a prompt template can be combined with the text to be searched, and the prompt template and the text to be searched can be input into the large language model to obtain the keywords in the text and the weights of each keyword. The prompt template is used to guide the large language model in identifying keywords and assigning weights to them.

[0089] In this embodiment, the prompt template can specifically indicate the search intent. For example, for the text "a beautiful sunset scene with rolling hills and a tranquil lake in the foreground," the prompt template could be: Search for natural landscape images, emphasizing "sunset" and "lake." By inputting the prompt template and text into a large language model, the large language model can determine keywords and assign weights based on the semantic understanding of the input text and the search intent indicated by the prompt template.

[0090] Furthermore, the suggestion template can also be a complete example of determining keywords and weights based on semantic analysis. The suggestion template can include the search intent, search focus, keywords, keyword weight allocation, and the rationale behind the allocation, all based on the input text. For example, the suggestion template content is as follows: Input: A beautiful sunset scene with rolling mountains and a tranquil lake in the foreground. Output: 1. Search intent: Search for natural landscape images, emphasizing "sunset" and "lake"; 2. Search focus: "sunset" and "lake" should be prioritized; 3. Keyword / phrase breakdown: "sunset," "mountains," "lake," "foreground"; 4. Weight combination: "sunset": 0.9, "lake": 0.8, "mountains": 0.6, "foreground": 0.4; 5. Rationale for weight allocation: "sunset" and "lake" are core elements and are given high weight; "mountains" are auxiliary elements in the scene and have medium priority; "foreground" is a supplementary description of the scene and has lower priority. The prompt template can also be filled with the text of the image to be searched. For example, the prompt template may include the following statement: "Now please complete the task based on the following input:". If "a man with a red suitcase and a pink hat" is added after this statement, the prompt template is input into the large language model. The large language model can learn the internal logic from the input text to the determination of keywords and weights based on the complete example in the prompt template, and determine the keywords and weights of the text of the current image to be searched by combining the semantic understanding results of the current image to be searched.

[0091] In step S302, the electronic device performs image-text matching for each keyword and determines the similarity between each keyword and each candidate image in the candidate image set. During image-text matching, the aforementioned CLIP model can be used, which extracts image feature vectors and text feature vectors and compares their similarity to determine the similarity between each keyword and each candidate image in the candidate image set; alternatively, a neural network method can be used, but this embodiment does not limit the specific approach.

[0092] In step S303, the electronic device determines the fusion similarity between each candidate image and all keywords based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword. Here, one candidate image corresponds to one fusion similarity, and the fusion similarity represents the overall similarity between the candidate image and all keywords in the text.

[0093] In determining the fusion similarity in this application embodiment, for example, the similarity between the candidate image and the keyword can be weighted using the keyword weights, and the weighted similarity results between the candidate image and each keyword can be fused (e.g., added together) to obtain the fusion similarity between the candidate image and all keywords. Of course, this application embodiment can also process the similarity between the candidate image and each keyword before weighting and fusing, or process the weighted similarity before fusing, or further process the weighted and fused similarity to obtain the final fusion similarity. This application embodiment does not limit this approach. In this application, both the similarity and weight are values ​​less than 1.

[0094] In step S304, the electronic device filters out target images that match the text from the candidate image set based on the fusion similarity of each candidate image. In some embodiments, the electronic device may directly use candidate images with high fusion similarity as target images; in other embodiments, the electronic device may also analyze the content of each candidate image and then combine the fusion similarity of the candidate images to comprehensively determine the target image.

[0095] It is understandable that in this embodiment, the electronic device first performs semantic analysis on the text. On the one hand, after determining the keywords, image-text matching is performed on each keyword, ensuring that every core keyword (semantic meaning) in the text is considered. Compared to directly performing image-text matching on the entire text, this reduces the likelihood of some core words being weakened during the overall text matching process. On the other hand, based on semantic analysis, weights are assigned to each keyword. After performing image-text matching on each keyword, the fusion similarity between each candidate image and all keywords is determined based on the weights of each keyword. The target image is then selected based on the fusion similarity. Compared to directly performing word segmentation matching on the text without considering weights, this application can highlight the dominant role of core semantics through weights while ensuring comprehensive matching (covering all keywords), thereby helping to select target images that better match the text from the candidate image set. The keyword-by-keyword matching in this embodiment, with weighted fusion similarity used to guide the selection process, improves the accuracy of the image-text matching results.

[0096] Furthermore, in this embodiment, the method of one semantic analysis and multiple image-text matching can effectively reduce the amount of computation and improve efficiency compared to the method that requires multiple semantic analyses and multiple image-text matchings of the text. Moreover, the semantic analysis and image-text matching functions in this application can be decoupled, and the lightweight decoupled architecture is also convenient to deploy in edge devices or user terminals.

[0097] In some embodiments, determining the fusion similarity between each candidate image and all keywords based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword, includes:

[0098] A non-linear mapping is performed on the similarity between each candidate image and each keyword to obtain a similarity mapping value between each candidate image and each keyword; wherein, if the similarity before mapping is greater than or equal to a first preset similarity threshold, the similarity mapping value after mapping is greater than the similarity; if the similarity is less than the first preset similarity threshold, the similarity mapping value is less than the similarity; the similarity mapping value is positively correlated with the similarity.

[0099] The similarity mapping values ​​between candidate images and keywords are weighted using the weight of keywords to obtain the weighted similarity between each candidate image and each keyword;

[0100] For each candidate image, the weighted similarity with each keyword is summed to obtain the fusion similarity between the candidate image and all keywords.

[0101] In this embodiment, the electronic device first performs a non-linear mapping on the similarity between candidate images and keywords. The purpose of the non-linear mapping is to make the similarity of large similarities larger and the similarity of small similarities smaller. The first preset similarity threshold used to distinguish the direction of the similarity mapping is, for example, 0.6. The following formula (1) is an example of the function used for the non-linear mapping:

[0102] (1)

[0103] It should be noted that the embodiments of this application are not limited to this kind of segmented mapping function, nor are they limited to the first preset similarity threshold of 0.6. Any method that can achieve the above-mentioned nonlinear mapping purpose is within the protection scope of this application.

[0104] In this embodiment, after performing a non-linear mapping on the similarity, the electronic device further weights the similarity mapping value using the weights of the keywords, and fuses the similarities of different keywords to obtain the fused similarity between the candidate image and all keywords. The following formula (2) is the formula for calculating the fused similarity:

[0105] (2)

[0106] in, Let the weight of the j-th keyword be . Let be the similarity mapping value between the candidate image and the j-th keyword. Let be the fusion similarity of the i-th candidate image.

[0107] Understandably, in this embodiment, the nonlinear mapping makes the similarity of high partitions quickly approach 1, which can strengthen the discrimination advantage of keywords that are more related to the candidate image; the accelerated decay of similarity in low partitions can effectively suppress the interference of keywords that are not strongly related to the candidate image; on this basis, the semantic weight of keywords in the text is further used to weight and fuse them, so that the fused similarity can take into account both the local feature contribution of each keyword and its contribution to the global semantics in the text, thus improving the accuracy of the target image obtained based on the fused similarity.

[0108] In some embodiments, the step of filtering target images that match the text from the candidate image set based on the fusion similarity of each candidate image includes:

[0109] Cluster analysis is performed on the candidate images based on their content to determine the clusters.

[0110] Based on the fusion similarity of candidate images in each cluster, the fusion similarity statistics corresponding to the cluster are determined.

[0111] Based on the fusion similarity statistics of each cluster, target images that match the text are selected.

[0112] In this embodiment, the electronic device first clusters candidate images based on their image content. During clustering, the clustering can be guided by the intra-cluster variance, with the goal of ensuring that the total intra-cluster variance of each cluster is less than a preset variance threshold. For example, within a preset maximum number of clusters, the clusters are determined with the goal of minimizing the total intra-cluster variance of all clusters.

[0113] For example, this application can determine the intra-class variance based on the following formula (3):

[0114] (3)

[0115] in, For the i-th cluster, For its center of mass, This is the image feature vector. Let be the total within-class variance of the k clusters.

[0116] In this embodiment, clustering methods such as K-means can be used. After determining the clusters, the electronic device further determines the fusion similarity statistics corresponding to the clusters based on the fusion similarity of candidate images in the clusters. For example, the fusion similarity statistics can be the median fusion similarity, the average fusion similarity, the maximum fusion similarity, etc. It can be understood that the fusion similarity statistics represent the relevance between the candidate images and the text in this cluster.

[0117] In this application, the electronic device further filters target images that match the text based on the fusion similarity statistics of each cluster. In some embodiments, the target images can be filtered directly based on the magnitude of the fusion similarity statistics of each cluster. For example, the electronic device selects the image in the cluster with the largest fusion similarity statistics as the target image that matches the text. In other embodiments, the electronic device can also filter target images based on the fusion similarity statistics of each cluster combined with the fusion similarity of candidate images within the cluster. For example, in clusters with fusion similarity statistics greater than a first threshold, candidate images with fusion similarity greater than a second threshold are selected as target images.

[0118] It is understood that in this embodiment of the application, the electronic device further performs clustering based on the image content (image semantics) of each candidate image, and determines the fusion similarity statistical value of the cluster based on the fusion similarity of each candidate image in the cluster to filter out the target image that matches the text. This can reduce the possibility that candidate images with the same semantics will be scattered due to differences in fusion similarity caused by matching each keyword in the text. By clustering candidate images with the same semantics and then filtering based on the fusion similarity statistical value, it is easier to filter out target images with high matching pairs between the image and each keyword in the text, and with high correlation between the complete image semantics and the text semantics, thereby improving the accuracy of target image filtering.

[0119] Furthermore, this application selects images from the clusters with the highest similarity statistics as target images for text matching, thus balancing the accuracy and efficiency of target image determination.

[0120] In some embodiments, the step of performing cluster analysis on candidate images based on the image content of each candidate image to determine clusters includes:

[0121] Cluster analysis is performed on the image content of each candidate image whose fusion similarity is greater than a preset second similarity threshold to determine the clusters.

[0122] In this embodiment, candidate images with low fusion similarity are essentially images that are not very relevant to the text. Therefore, the electronic device can filter based on the fusion similarity and select candidate images with a similarity greater than or equal to a second preset similarity threshold to participate in subsequent clustering, which can reduce the amount of computation and improve the efficiency of target image filtering.

[0123] In some embodiments, the step of performing cluster analysis on candidate images based on the image content of each candidate image to determine clusters includes:

[0124] The number of clusters K is preset. Clustering is performed based on the image content of each candidate image, and the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters is determined.

[0125] In response to the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters, the decrease in the total intra-class variance of the K-1 clusters relative to the total intra-class variance of the K-2 clusters becomes smaller, and the difference between the two is less than a preset difference threshold, the K clusters are determined as the clusters.

[0126] In this embodiment, a preset number of clusters K is used. The electronic device determines the total intra-cluster variance when there are K clusters and the total intra-cluster variance when there are K-1 clusters, and compares the decrease in the total intra-cluster variance of the K clusters relative to that of the K-1 clusters. When the Kth cluster is added, the total intra-cluster variance decreases significantly, indicating that introducing the Kth cluster is reasonable. However, if the improvement in the total intra-cluster variance from adding the Kth cluster is negligible, it may be forcibly splitting already compact clusters, and the introduction of the Kth cluster is actually unnecessary. It is understandable that the decrease in the total intra-cluster variance can be used to quantify the "benefit" of adding the Kth cluster.

[0127] In this embodiment, if the "benefit" of adding the Kth cluster is smaller than the "benefit" of adding the (K-1)th cluster (the decrease in the total intra-class variance of the K-1 clusters relative to the decrease in the total intra-class variance of the K-2 clusters), and the difference between the two is less than a preset difference threshold, then it indicates that adding the Kth cluster is the turning point where the benefit decreases sharply, and the Kth clusters are also the optimal clusters. It should be noted that in this embodiment, the most suitable value of K can be found by gradually increasing the number of clusters K and observing the decrease. For example, the optimal value of K can be found by referring to the following formula (4):

[0128] (4)

[0129] in, A preset difference threshold can be set, with a value of 0.5. The decrease in the total within-class variance of the K clusters relative to the total within-class variance of the K-1 clusters; This represents the decrease in the total within-class variance of the K-1 clusters relative to the total within-class variance of the K-2 clusters.

[0130] Understandably, this application achieves a critical point where the overall intra-class variance decreases significantly for the first time, reducing the arbitrariness and uncertainty of human judgment. It can determine the optimal number of clusters more accurately under unsupervised conditions, while taking into account intra-class compactness and reducing overfitting problems caused by too many clusters.

[0131] In some embodiments, determining the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters includes:

[0132] The difference between the intra-class variance when all target candidate images are clustered into one class and the total intra-class variance when all target candidate images are clustered into a preset maximum number of classes is used as the denominator. The difference between the total intra-class variance of K-1 clusters and the total intra-class variance of K clusters is used as the numerator. The decrease magnitude is determined based on the ratio of the numerator to the denominator.

[0133] In this embodiment of the application, the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters can be referred to the following formula (5):

[0134] (5)

[0135] in, The within-class variance when all samples are considered as a single cluster. The total intra-class variance when the maximum number of clusters is preset, for example, the maximum number of clusters is 10.

[0136] It is understandable that the above calculation method can uniformly map the decrease in the number of clusters under different clusters to a comparable scale of [0,1], providing a stable and reliable quantitative basis for objectively determining the best cluster.

[0137] In some embodiments, determining the fusion similarity between each candidate image and all keywords based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword, includes:

[0138] From the candidate image set, candidate images whose similarity to the target keyword is greater than a third preset similarity threshold are selected; wherein, the target keyword is a keyword with a weight greater than a preset weight threshold;

[0139] For each selected candidate image, the fusion similarity between each candidate image and all keywords is determined based on the similarity between the candidate image and each keyword, as well as the weight of the keyword.

[0140] In this embodiment, the target keywords with a weight greater than a preset weight threshold are, for example, one or more keywords with the highest weight. Since target keywords have a greater semantic impact on the text, if a candidate image has a low similarity to each target keyword, it indicates that the candidate image is unlikely to match the text. Therefore, such candidate images can be filtered out, while candidate images with a similarity to the target keywords greater than a third preset similarity threshold are retained for subsequent fusion similarity calculation.

[0141] For example, if the similarity between a candidate image and the top 2 keywords (sunset, lake) in a long sentence is less than 0.5, then the result of this candidate image will not be included in the subsequent calculation.

[0142] It should be noted that, in this embodiment of the application, the selection can also be based on the average similarity between the candidate image and all target keywords, and no limitation is imposed here.

[0143] It is understood that in this embodiment of the application, pre-screening a portion of candidate images based on their similarity to the target keywords helps reduce the amount of subsequent computation and reduces the interference of irrelevant candidate images on the overall result in the subsequent fusion and sorting stage, thereby improving the efficiency and accuracy of target image determination.

[0144] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario. See also... Figure 4 , Figure 4 This is a flowchart illustrating another image-text matching method provided in an embodiment of this application. For example, when a user enters text in a search box, target images matching the text can be quickly and accurately filtered from the image library based on the following steps. Figure 4 The steps outlined in this application will be used to describe the image-text matching method provided in the embodiments.

[0145] S401, Long text is embedded in the prompt template.

[0146] In this embodiment of the application, the long text embedded in the prompt template is the text to be searched.

[0147] S402, Large Language Model Processing Prompt Template.

[0148] In this embodiment, the large language model processes the prompt template by learning the inherent logic from the input text to the determination of keywords and weights based on the complete examples in the prompt template, and performs semantic understanding on the text to be searched in order to determine the keywords and weights of the current text.

[0149] S403, The output of the analytical model is a weighted phrase group.

[0150] In this embodiment of the application, each phrase in the phrase group is a keyword.

[0151] S404, The image-text matching model processes input data.

[0152] In this embodiment of the application, the input data for the image-text matching model is a phrase group, and the image-text matching model is, for example, the CLIP model.

[0153] S405. Output phrase and image similarity > 0.6? If not, proceed to step S406; if yes, proceed to step S407.

[0154] In this embodiment, the similarity between each phrase and each image (candidate image) can be obtained through the image-text matching model. Here, 0.6 represents the first preset similarity threshold. Based on the comparison result between the similarity and the first preset similarity threshold, different non-linear mapping methods are executed.

[0155] S406, First method to optimize similarity.

[0156] In this embodiment of the application, if the similarity is greater than the first preset similarity threshold, then based on the above formula (1) Optimize similarity.

[0157] S407, Second method to optimize similarity.

[0158] In this embodiment of the application, if the similarity is less than the first preset similarity threshold, then based on the above formula (1) Optimize similarity.

[0159] S408. The final image score is calculated by combining similarity with weights.

[0160] In this embodiment of the application, the similarity combined with the weight is used to calculate the final image score, which is to calculate the fusion similarity of the candidate images.

[0161] S409. Perform clustering operations on the top K images with the highest scores.

[0162] In this embodiment, the Top K images are the candidate images whose fusion similarity is greater than a preset second similarity threshold.

[0163] S410. Output the search results by combining image clustering and scoring results.

[0164] In this embodiment of the application, the retrieval results output by combining image clustering and scoring results can refer to the aforementioned fusion similarity statistics based on each cluster to filter out target images that match the text.

[0165] It is understood that the embodiments of this application can improve the efficiency of image-text matching and the accuracy of image-text matching results.

[0166] The following description continues to illustrate the exemplary structure of the image-text matching device 255 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the image-text matching device 255 in the memory 250 may include:

[0167] The semantic analysis module 2551 is configured to perform semantic analysis on the text to determine the keywords in the text and the weight of each keyword's semantic influence.

[0168] The image-text matching module 2552 is configured to perform image-text matching for each keyword and determine the similarity between each keyword and each candidate image in the candidate image set.

[0169] The fusion similarity determination module 2553 is configured to determine the fusion similarity between each candidate image and all keywords based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword.

[0170] The filtering module 2554 is configured to filter out target images that match the text from the candidate image set based on the fusion similarity of each candidate image.

[0171] In some embodiments, the fusion similarity determination module is further configured to perform a non-linear mapping on the similarity between each candidate image and each keyword to obtain a similarity mapping value between each candidate image and each keyword; wherein, if the similarity before mapping is greater than or equal to a first preset similarity threshold, the similarity mapping value after mapping is greater than the similarity; if the similarity is less than the first preset similarity threshold, the similarity mapping value is less than the similarity; the similarity mapping value is positively correlated with the similarity; the similarity mapping values ​​between candidate images and keywords are weighted using the weights of keywords to obtain a weighted similarity between each candidate image and each keyword; for each candidate image, the weighted similarity with each keyword is added to obtain the fusion similarity between the candidate image and all keywords.

[0172] In some embodiments, the filtering module is further configured to perform cluster analysis on the candidate images based on the image content of each candidate image to determine clusters; determine the fusion similarity statistics of the clusters based on the fusion similarity of the candidate images in each cluster; and filter out target images that match the text based on the fusion similarity statistics of each cluster.

[0173] In some embodiments, the filtering module is further configured to preset the number of clusters K, perform clustering based on the image content of each candidate image, and determine the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters; in response to the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters being smaller than the decrease in the total intra-class variance of the K-2 clusters relative to the total intra-class variance of the K-1 clusters, and the difference between the two being less than a preset difference threshold, the K clusters are determined as the clusters.

[0174] In some embodiments, the filtering module is further configured to use the difference between the intra-class variance when all target candidate images are clustered into one class and the total intra-class variance when all target candidate images are clustered into a preset maximum number of classes as the denominator, and the difference between the total intra-class variance of K-1 clusters and the total intra-class variance of K clusters as the numerator, and determine the decrease magnitude based on the ratio of the numerator and the denominator.

[0175] In some embodiments, the filtering module is further configured to perform cluster analysis on the candidate images based on the image content of each candidate image whose fusion similarity is greater than a preset second similarity threshold, and determine the cluster.

[0176] In some embodiments, the filtering module is further configured to select the image from the cluster with the highest fusion similarity statistical value among all clusters as the target image to match the text.

[0177] In some embodiments, the fusion similarity determination module is further configured to select candidate images from the candidate image set whose similarity to the target keyword is greater than a third preset similarity threshold; wherein the target keyword is a keyword with a weight greater than a preset weight threshold; and for each selected candidate image, the fusion similarity between each candidate image and all keywords is determined based on the similarity between the candidate image and each keyword and the weight of the keyword.

[0178] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the electronic device to perform the method described in this application.

[0179] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions. When the computer program or executable instructions are executed by a processor, the processor will execute the image-text matching method provided in this application. For example, ... Figure 3 The image-text matching method is shown.

[0180] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0181] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0182] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0183] As an example, computer-executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0184] In summary, the embodiments of this application can save computing power and reduce power consumption while minimizing the loss of prediction accuracy.

[0185] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for matching images and text, characterized in that, The method includes: Perform semantic analysis on the text to determine the keywords in the text and the weight of each keyword's semantic influence; Perform image-text matching for each keyword and determine the similarity between each keyword and each candidate image in the candidate image set; Based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword, the fusion similarity between each candidate image and all keywords is determined. Based on the fusion similarity of each candidate image, target images that match the text are selected from the candidate image set.

2. The method according to claim 1, characterized in that, The determination of the fusion similarity between each candidate image and all keywords, based on the similarity between each keyword and each candidate image in the candidate image set, and the weight of the keyword, includes: A non-linear mapping is performed on the similarity between each candidate image and each keyword to obtain a similarity mapping value between each candidate image and each keyword; wherein, if the similarity before mapping is greater than or equal to a first preset similarity threshold, the similarity mapping value after mapping is greater than the similarity; if the similarity is less than the first preset similarity threshold, the similarity mapping value is less than the similarity; the similarity mapping value is positively correlated with the similarity. The similarity mapping values ​​between candidate images and keywords are weighted using the weight of keywords to obtain the weighted similarity between each candidate image and each keyword; For each candidate image, the weighted similarity with each keyword is summed to obtain the fusion similarity between the candidate image and all keywords.

3. The method according to claim 1, characterized in that, The step of selecting target images that match the text from the candidate image set based on the fusion similarity of each candidate image includes: Cluster analysis is performed on the candidate images based on their content to determine the clusters. Based on the fusion similarity of candidate images in each cluster, the fusion similarity statistics corresponding to the cluster are determined. Based on the fusion similarity statistics of each cluster, target images that match the text are selected.

4. The method according to claim 3, characterized in that, The step of performing cluster analysis on candidate images based on their content to determine clusters includes: The number of clusters K is preset. Clustering is performed based on the image content of each candidate image, and the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters is determined. In response to the decrease in the total intra-class variance of the K clusters relative to the total intra-class variance of the K-1 clusters, the decrease in the total intra-class variance of the K-1 clusters relative to the total intra-class variance of the K-2 clusters becomes smaller, and the difference between the two is less than a preset difference threshold, the K clusters are determined as the clusters.

5. The method according to claim 4, characterized in that, The determination of the decrease in the total within-class variance of the K clusters relative to the total within-class variance of the K-1 clusters includes: The difference between the intra-class variance when all target candidate images are clustered into one class and the total intra-class variance when all target candidate images are clustered into a preset maximum number of classes is used as the denominator. The difference between the total intra-class variance of K-1 clusters and the total intra-class variance of K clusters is used as the numerator. The decrease magnitude is determined based on the ratio of the numerator to the denominator.

6. The method according to claim 3, characterized in that, The step of performing cluster analysis on candidate images based on their content to determine clusters includes: Cluster analysis is performed on candidate images based on their content, where the fusion similarity is greater than a preset second similarity threshold, to determine the clusters.

7. The method according to claim 3, characterized in that, The process of filtering target images that match the text based on the fusion similarity statistics of each cluster includes: The image from the cluster with the highest fusion similarity score among all clusters is selected as the target image for matching the text.

8. The method according to any one of claims 1 to 7, characterized in that, The determination of the fusion similarity between each candidate image and all keywords, based on the similarity between each keyword and each candidate image in the candidate image set, and the weight of the keyword, includes: From the candidate image set, candidate images whose similarity to the target keyword is greater than a third preset similarity threshold are selected; wherein, the target keyword is a keyword with a weight greater than a preset weight threshold; For each selected candidate image, the fusion similarity between each candidate image and all keywords is determined based on the similarity between the candidate image and each keyword, as well as the weight of the keyword.

9. A text-image matching device, characterized in that, The device includes: The semantic analysis module is configured to perform semantic analysis on the text to determine the keywords in the text and the weight of each keyword's semantic influence. The image-text matching module is configured to perform image-text matching for each keyword and determine the similarity between each keyword and each candidate image in the candidate image set. The fusion similarity determination module is configured to determine the fusion similarity between each candidate image and all keywords based on the similarity between each keyword and each candidate image in the candidate image set, as well as the weight of the keyword. The filtering module is configured to filter out target images that match the text from the candidate image set based on the fusion similarity of each candidate image.

10. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images