Multi-language supported text and image searching method and device, storage medium and computer equipment

Abstract

According to the multi-language-supporting image searching method and device, the storage medium and the computer equipment, the query text of any language input by the user is received, the query text is translated into the English text by using the advanced translation model, and the English text is encoded into the query vector, so that multi-language query is supported. Meanwhile, according to the method, a vector database containing image description vectors is also constructed, and candidate images are reordered and screened by calculating similarity scores between query vectors and the image description vectors and keyword matching scores between English texts and English description texts, so that the semantic correlation is ensured, and the accuracy of the candidate images is improved. And the hit rate of the keywords is ensured, and finally an image set highly matched with the query intention of the user is output. The process not only improves the accuracy and efficiency of searching the image by the text, but also greatly widens the application range of the user, and meets the information retrieval requirement under the multi-language environment.

Description

Technical Field

[0001] This application relates to the field of image and text retrieval technology, and in particular to a text and image search method, apparatus, storage medium and computer equipment that supports multiple languages. Background Technology

[0002] In the current field of image and text retrieval technology, traditional text-to-image search methods often only support queries in a single language, which greatly limits the user experience and search scope. With the acceleration of globalization, the demand for information retrieval in multilingual environments is growing, and users hope to be able to search in their familiar language and obtain accurate and relevant image results. However, existing technical solutions, when handling multilingual queries, either require users to manually translate the query text into a language supported by the system, or the system can only process queries in a specific language. This undoubtedly increases the user's burden and reduces search efficiency.

[0003] Furthermore, existing image-text retrieval processes rely solely on vector similarity, making it difficult to fully consider semantic differences between languages ​​when handling multilingual queries, thus affecting the accuracy and relevance of search results. In addition, while relying solely on vector similarity can reflect the correlation between the query text and the image description to some extent, it ignores the direct correspondence between keywords in the text and specific content in the image description. This limitation is particularly pronounced when dealing with complex queries or polysemous words. Summary of the Invention

[0004] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the technical shortcomings of existing text search image methods that only support queries in a single language, which greatly limits the user experience and search scope, and also results in low retrieval accuracy and relevance.

[0005] This application provides a text search image method that supports multiple languages, the method comprising:

[0006] Receive query text in any language input by the user, translate the query text into English text, and encode the English text into a query vector;

[0007] Search a pre-built vector database for multiple candidate images similar to the query vector, along with their corresponding English descriptive text and image description vectors;

[0008] Based on the similarity score between the query vector and each of the image description vectors, and the keyword matching score between the English text and each of the English description texts, the candidate images are reordered and filtered to form a candidate set;

[0009] Based on the English text and the English description text corresponding to each candidate image in the candidate set, the candidate images in the candidate set are finely ranked and semantically verified, and the final image set is output.

[0010] Optionally, translating the query text into English text includes:

[0011] The query text is translated by calling a translation model that optimizes the search intent and supports multiple languages, and the length of the translated text is controlled within a preset length range to obtain the English text output by the translation model.

[0012] Optionally, the process of constructing the vector database includes:

[0013] Retrieve all images from the image library;

[0014] A visual language model guided by structured cue words generates English descriptive text corresponding to each image. Each English descriptive text includes at least the object, scene, and action of the corresponding image.

[0015] The English text descriptions are encoded into image description vectors using a unified English text encoder, and each image description vector and its associated metadata are saved to a vector database.

[0016] Optionally, the step of reordering and filtering the candidate images based on the similarity score between the query vector and each of the image description vectors, and the keyword matching score between the English text and each of the English description texts, to form a candidate set, includes:

[0017] Calculate the similarity score between the query vector and each of the image description vectors, and the keyword matching score between the English text and each of the English description texts;

[0018] The total score for each candidate image is determined based on the similarity score and the keyword matching score.

[0019] Based on the total score of each candidate image, the candidate images are reordered and filtered to form a candidate set.

[0020] Optionally, calculating the keyword matching scores between the English text and each of the English description texts includes:

[0021] Each structured field in the English description text is concatenated into a complete text document;

[0022] Each text document and the English text are preprocessed to obtain multiple preprocessed text documents and preprocessed English text.

[0023] Calculate the BM25 score between the preprocessed English text and each of the preprocessed text documents, and use the BM25 score as the keyword matching score.

[0024] Optionally, the step of refining and semantically verifying each candidate image in the candidate set based on the English text and the corresponding English description text of each candidate image in the candidate set, and outputting the final image set, includes:

[0025] The English text and the English description text corresponding to each candidate image in the candidate set are input into a preset scoring model to obtain the relevance score between the English text and each English description text in the candidate set, as output by the scoring model.

[0026] The candidate images in the candidate set are ranked according to their relevance scores to obtain the sorted candidate set.

[0027] Semantic verification is performed on each candidate image in the sorted candidate set, and the candidate images in the sorted candidate set are filtered according to the semantic verification results to output the final image set.

[0028] Optionally, the step of performing semantic verification on each candidate image in the sorted candidate set, and filtering each candidate image in the sorted candidate set according to the semantic verification results to output the final image set includes:

[0029] Using a preset prompt word template, the English description text corresponding to each candidate image in the sorted candidate set is combined with the English text to form multiple questions to be verified.

[0030] Each of the questions to be verified is input into the large language model for reasoning and judgment, and the judgment results output by the large language model corresponding to each of the questions to be verified are obtained.

[0031] Based on each of the aforementioned decision results, the candidate images in the sorted candidate set are filtered to output the final image set.

[0032] This application also provides a text search image device that supports multiple languages, including:

[0033] The translation and encoding module is used to receive query text in any language input by the user, translate the query text into English text, and encode the English text into a query vector;

[0034] The similarity search module is used to search in a pre-built vector database for multiple candidate images similar to the query vector, along with their corresponding English descriptive text and image description vectors.

[0035] The image filtering module is used to reorder and filter each candidate image based on the similarity score between the query vector and each of the image description vectors, and the keyword matching score between the English text and each of the English description texts, to form a candidate set;

[0036] The image reordering module is used to perform fine sorting and semantic verification on each candidate image in the candidate set based on the English text and the English description text corresponding to each candidate image in the candidate set, and output the final image set.

[0037] This application also provides a computer-readable storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the multilingual text search image method as described in any of the above embodiments.

[0038] This application also provides a computer device, including: one or more processors, and memory;

[0039] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the multilingual text search image method as described in any of the above embodiments.

[0040] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0041] This application provides a multilingual text search image method, apparatus, storage medium, and computer device. The method receives query text in any language input by the user, translates it into English text using an advanced translation model, and encodes the English text into query vectors, thus supporting multilingual queries. Simultaneously, the method constructs a vector database containing image description vectors. By calculating the similarity score between the query vector and the image description vector, and the keyword matching score between the English text and the English description text, candidate images are reordered and filtered. This ensures both semantic relevance and keyword hit rate, ultimately outputting an image set highly matched to the user's query intent. This process not only improves the accuracy and efficiency of text search image search but also greatly expands the user's scope, meeting the information retrieval needs in a multilingual environment. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a text search image method supporting multiple languages ​​provided in this application embodiment;

[0044] Figure 2 A schematic diagram illustrating the process of reordering and filtering candidate images provided in an embodiment of this application;

[0045] Figure 3 A schematic diagram illustrating the process of refining and semantically verifying each candidate image in the candidate set, as provided in an embodiment of this application;

[0046] Figure 4 A schematic diagram of the structure of a text search image device supporting multiple languages ​​provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] In one embodiment, such as Figure 1 As shown, Figure 1 This application provides a flowchart illustrating a text search image method supporting multiple languages, as illustrated in an embodiment of the present application. The present application provides a text search image method supporting multiple languages, which may include:

[0050] S110: Receives query text in any language input by the user, translates the query text into English text, and encodes the English text into a query vector.

[0051] In this step, the image and text retrieval system can activate a multilingual receiving interface that is compatible with and recognizes many mainstream languages ​​worldwide, such as Chinese, French, Spanish, and Arabic. When the user enters query text in any language, the system quickly invokes its built-in translation model. This model, trained on extensive multilingual data, possesses high-precision translation capabilities and can accurately translate the user's query text into English.

[0052] After translation, the system can also use a unified English text encoder to encode the English text into query vectors. This encoding method can transform the semantic information of the text into a numerical vector form that computers can understand and process, laying the foundation for subsequent similarity searches.

[0053] S120: Search a pre-built vector database for multiple candidate images similar to the query vector, along with their corresponding English descriptive text and image descriptive vectors.

[0054] In this step, after translating the user-input query text into English text through S110 and encoding the English text into a query vector, this application can also search for multiple candidate images similar to the query vector and their corresponding English description text and image description vectors in a pre-built vector database. In this way, each candidate image can be filtered based on its English description text and image description vector.

[0055] The vector database pre-constructed in this application refers to a carefully organized and optimized dataset containing a large number of images along with corresponding English descriptive text and image description vectors. For example, this application can utilize a structured cue-guided visual language model to generate corresponding English descriptive text for each image in the image library. These descriptive texts not only cover the main objects in the image but also describe in detail the scene, actions, emotions, atmosphere, artistic style, and shooting style, thus providing rich semantic information.

[0056] Subsequently, this application uses an English text encoder to encode these English descriptive texts into image description vectors, which accurately represent the semantic content of the descriptive texts. Finally, this application can save these image description vectors and related metadata, such as image ID, source, and resolution, to a vector database for subsequent similarity search and retrieval operations. Through this construction process, the vector database can provide efficient and accurate image retrieval services, support multilingual query text input, and return an image set that highly matches the query intent.

[0057] In practical applications, this application allows the query vector to be input into a vector database. The vector database quickly initiates a search mechanism, calculating the similarity between the query vector and various image description vectors in the database to identify the multiple image description vectors most similar to the query vector. The images corresponding to these similar image description vectors are the candidate images. Simultaneously, the system also obtains the corresponding English descriptive text for these candidate images, enabling more precise filtering and sorting later. This process fully leverages the efficient search capabilities of the vector database, quickly filtering candidate images from a massive amount of images that most closely match the user's query intent, providing strong support for subsequent search results.

[0058] S130: Based on the similarity score between the query vector and each image description vector, and the keyword matching score between the English text and each English description text, the candidate images are reordered and filtered to form a candidate set.

[0059] In this step, after searching for multiple candidate images similar to the query vector and their corresponding English description text and image description vectors through S120, this application can reorder and filter each candidate image based on the similarity score between the query vector and each image description vector, as well as the keyword matching score between the English text and each English description text, thereby forming a candidate set.

[0060] Specifically, after obtaining multiple candidate images most similar to the query vector, along with their corresponding English descriptive text and image descriptive vectors, this application further refines the candidate images by calculating similarity scores between the query vector and each image descriptive vector. This step utilizes a vector space model, calculating cosine similarity or other similarity metrics to quantify the semantic similarity between the query vector and the image descriptive vectors. Simultaneously, this application can also calculate keyword matching scores between the English text and each English descriptive text. This step uses natural language processing techniques to identify and match keywords in the query text and descriptive text to assess their content relevance.

[0061] After obtaining the similarity score and keyword matching score of each candidate image, this application can comprehensively consider these scores to determine the total score of each candidate image. This process can be achieved through weighted summation or other fusion strategies to ensure that both the similarity score and the keyword matching score have a reasonable impact on the final total score.

[0062] Finally, this application can reorder and filter candidate images based on their total scores. For example, this application can sort candidate images in descending order of total score and filter out candidate images with total scores exceeding a certain threshold to form a candidate set. In this way, the images in the candidate set will be more likely to highly match the user's query intent, thereby improving the accuracy and efficiency of the retrieval.

[0063] S140: Based on the English text and the English description text corresponding to each candidate image in the candidate set, perform fine sorting and semantic verification on each candidate image in the candidate set, and output the final image set.

[0064] In this step, after reordering and filtering each candidate image in S130 to form a candidate set, this application can further refine and semantically verify each candidate image in the candidate set based on the English text and the English description text corresponding to each candidate image in the candidate set, thereby obtaining the final image set.

[0065] Understandably, although the candidate set has already undergone one round of screening and sorting, this application can further refine the sorting and semantic-level verification to improve the accuracy and reliability of the retrieval. Specifically, when refining the ranking of each candidate image in the candidate set, this application can employ various methods. For example, this application can input the English text and the corresponding English descriptive text of each candidate image in the candidate set into a pre-defined scoring model to score relevance. This application can also employ other deep learning-based ranking algorithms, such as Learning to Rank models, to more accurately rank the candidate images. These scoring models or ranking algorithms are typically trained on a large amount of labeled data and can capture the complex semantic relationships between text and images, thereby providing a more accurate relevance assessment.

[0066] After obtaining the relevance scores of each candidate image, this application can reorder the candidate images in descending order of score to ensure that the images that best match the user's query intent are ranked first. After fine-tuning, this application can also perform semantic verification on the sorted candidate images. The purpose of this step is to further filter out images that, although they match the query text at the keyword level, have semantic deviations or inconsistencies.

[0067] To achieve this goal, this application can employ various semantic verification methods. For example, it can utilize pre-trained language models or image description generation models to perform semantic understanding and analysis on the English description text of candidate images, determining their semantic consistency with the query text. Alternatively, it can utilize visual semantic embedding technology in image retrieval to map images and text into the same semantic space, and perform verification by calculating the semantic distance between them. The specific method can be chosen based on the actual situation, and no restrictions are imposed here.

[0068] In the above embodiments, this method receives query text in any language input by the user, translates it into English text using an advanced translation model, and encodes the English text into query vectors, thereby supporting multilingual queries. Simultaneously, the method constructs a vector database containing image description vectors. By calculating the similarity score between the query vector and the image description vector, and the keyword matching score between the English text and the English description text, candidate images are reordered and filtered. This ensures both semantic relevance and keyword hit rate, ultimately outputting an image set highly matching the user's query intent. This process not only improves the accuracy and efficiency of text-to-image search but also greatly expands the user's scope, meeting the information retrieval needs in a multilingual environment.

[0069] In one embodiment, translating the query text into English text in S110 may include:

[0070] S111: Call a translation model that optimizes retrieval intent and supports multiple languages ​​to translate the query text, and control the length of the translated text within a preset length range to obtain the English text output by the translation model.

[0071] In this embodiment, to ensure the accuracy and practicality of the translation, this application specifically selects a translation model optimized for search intent and supporting multiple languages. This translation model not only possesses powerful multilingual translation capabilities, capable of handling translation tasks in multiple mainstream languages ​​worldwide, but is also specifically optimized for search scenarios, enabling it to better understand the user's query intent and thus provide translation results that better meet the user's needs.

[0072] For example, this application can choose a general model and optimize and train it at the retrieval intent level. During training, a large amount of retrieval query text covering different fields and expressions, along with their corresponding high-quality English translations, are collected as training data. Through deep learning on this data, the translation model learns how to accurately grasp user intent in image retrieval scenarios and precisely translate query text in various languages ​​into English.

[0073] Furthermore, considering factors such as subsequent encoding and search efficiency, this application can also control the length of the translated text within a preset length range during the translation process. This preset length range was determined through extensive experiments and data analysis, ensuring that the translated English text fully retains the key information of the original query text while avoiding excessive length that could affect the efficiency and accuracy of subsequent processing, ultimately resulting in a satisfactory English text.

[0074] In one embodiment, the process of constructing the vector database may include:

[0075] S210: Retrieve all images from the image library.

[0076] S211: Generate English descriptive text corresponding to each image based on a visual language model guided by structured cue words. Each English descriptive text includes at least the object, scene, and action of the corresponding image.

[0077] S212: Use a unified English text encoder to encode each English description text into an image description vector, and save each image description vector and related metadata to a vector database.

[0078] In this embodiment, the construction of the vector database is a crucial foundation for the efficient operation of the entire text search image system. After acquiring all images in the image library, this application can utilize a visual language model guided by structured prompts to generate detailed English descriptive text for each image. These descriptive texts not only accurately capture the main objects in the image but also deeply depict the scene layout, actions, and potential emotional atmosphere, providing rich semantic information for subsequent similarity searches.

[0079] In one specific implementation, this application can use a visual language model (such as Qwen2.5-VL, LLaVA, InternVL) to generate English descriptive text corresponding to each image, and enforce the use of structured cue words to guide the model to output English text containing the following five dimensions. Example:

[0080] Prompt template:

[0081] Describe this image in English with the following structure:

[0082] Object: [Main Object]

[0083] Scene: [Scene / Environment]

[0084] Action: [Character / Object Action]

[0085] Emotion / Atmosphere: [emotion / atmosphere]

[0086] Style: [Art / Photography Style]

[0087] Keep it detailed and over 50 words.

[0088] Output example:

[0089] Object: A construction worker wearing a red helmet and black boots.

[0090] Scene: Standing near steel scaffolding under bright daylight.

[0091] Action: Holding a white plastic bag in right hand, pointing withleft.

[0092] Emotion: Focused and determined.

[0093] Style: Realistic, high contrast, industrial photography.

[0094] Subsequently, this application employs a unified English text encoder (such as BGE-large-en-v1.5, E5-base) to transform these descriptive texts into 768 / 1024 image description vectors. These vectors accurately reflect the semantic content of the descriptive texts, laying the foundation for subsequent vector similarity calculations. Finally, this application can save these image description vectors along with image metadata, such as image ID, source, and resolution, into a vector database ( / PGVectorMilvus / Qdrant / FAISS), constructing a well-structured and richly detailed dataset that provides strong support for subsequent image retrieval.

[0095] In one embodiment, such as Figure 2 As shown, Figure 2This is a schematic diagram illustrating the process of reordering and filtering candidate images according to an embodiment of this application; in S130, after reordering and filtering the candidate images based on the similarity score between the query vector and each of the image description vectors, and the keyword matching score between the English text and each of the English description texts, a candidate set is formed, which may include:

[0096] S131: Calculate the similarity score between the query vector and each of the image description vectors, and the keyword matching score between the English text and each of the English description texts.

[0097] S132: Determine the total score for each candidate image based on the similarity score and the keyword matching score.

[0098] S133: Based on the total score of each candidate image, the candidate images are reordered and filtered to form a candidate set.

[0099] In this embodiment, when reordering and filtering candidate images, this application first calculates the similarity score between the query vector and each image description vector. This step relies on the vector space model, quantifying the semantic similarity between the query vector and the image description vector by calculating the cosine similarity or other metrics between vectors. Simultaneously, this application can also calculate the keyword matching score between the English text and each English description text. This step utilizes natural language processing technology to assess the content relevance of the query text and the English description text by identifying and matching keywords in them.

[0100] After obtaining the similarity score and keyword matching score of each candidate image, this application can comprehensively consider these scores to determine the total score of each candidate image. This comprehensive process can be achieved through weighted summation or other fusion strategies to ensure that both the similarity score and the keyword matching score have a reasonable and appropriate impact on the final total score.

[0101] In one specific implementation, this application can calculate the total score of each candidate image using the following formula, and the calculation process is as follows:

[0102]

[0103] Where Cosine_Similarity is the cosine similarity between the query vector and the image description vector, BM25_Score is the keyword matching score between the query text and the English description text, and λ is the weight.

[0104] Finally, based on the total score of each candidate image, this application can reorder them and filter out candidate images with a total score exceeding a preset threshold, thereby forming the final candidate set. This candidate set will be more likely to contain images that highly match the user's query intent, thus improving the accuracy and efficiency of the retrieval.

[0105] In one embodiment, calculating the keyword matching score between the English text and each of the English description texts in S131 may include:

[0106] S1311: Concatenate the structured fields in each of the English description texts into a complete text document.

[0107] S1312: Preprocess each text document and the English text to obtain multiple preprocessed text documents and preprocessed English text.

[0108] S1313: Calculate the BM25 score between the preprocessed English text and each of the preprocessed text documents, and use the BM25 score as the keyword matching score.

[0109] In this embodiment, to accurately calculate the keyword matching score between the English text and each English descriptive text, this application first concatenates the structured fields in each English descriptive text, such as object, scene, action, emotional atmosphere, and art style, into a complete text document. This ensures that all relevant information is taken into consideration, avoiding matching biases caused by missing information.

[0110] Taking the English description text above as an example, this application can concatenate it into the following text document:

[0111] "A construction worker wearing a red helmet and black boots. Standingnear steel scaffolding under bright daylight. Holding a white plastic bag inright hand, pointing with left. Focused and determined. Realistic, highcontrast, industrial photography."

[0112] Subsequently, this application preprocesses each concatenated text document and the English text entered by the user, including but not limited to word segmentation, removal of stop words (such as filtering out common but low-information words such as "a", "the", "in", "with"), stemming / lemma restoration, etc., in order to unify the text format, reduce noise interference, and improve the accuracy of subsequent calculations.

[0113] After preprocessing, this application can use the BM25 algorithm to calculate the relevance score between the preprocessed English text and each preprocessed text document. The BM25 algorithm is a classic ranking function that evaluates the similarity between query text and documents by considering factors such as word frequency, document length, and inverse document frequency, making it very suitable for calculating keyword matching scores.

[0114] Specifically, suppose the user query is: "construction worker pointing at something". The query text is segmented and preprocessed to obtain terms ["construction", "worker", "point", "something"]. For each term, this application can calculate its score in the aforementioned text document:

[0115] f("construction", D) = 1 (appears once);

[0116] f("worker", D) = 1;

[0117] f("point", D) = 1 (stemmed from "pointing");

[0118] f("something", D) = 0 (not found);

[0119] Calculate the IDF for each term (requires a document set based on the entire image database):

[0120] If "construction" is a relatively uncommon word, its IDF value will be very high.

[0121] Assume that "worker" is relatively common and its IDF value is moderate.

[0122] The IDF value of "point" is also moderate.

[0123] "something" is not included in the calculation since it does not appear.

[0124] Substitute into the formula: Substitute f(qi, D), DF(qi), document length |D|, average document length avgdl, parameter k1, and b into the BM25 formula for each term. The basic formula is as follows:

[0125] Score(D, Q) =Σ(i=1 to n) IDF(qi)*[ (f(qi, D)*(k1+1)) / (f(qi, D) + k1*(1-b+ b*|D| / avgdl))]

[0126] Where f(qi, D) is the frequency of term qi in document D, |D| is the length of document D (usually referring to the total number of terms), avgdl is the average length of all documents in the entire document set, k1 and b are free parameters, usually with default values ​​of k1=1.2-2.0 and b=0.75. k1 is used to control the saturation rate of term frequency; the larger the value, the stronger the effect of high-frequency words. b is used to control the degree of normalization of document length; b=1 is full normalization, and b=0 is no length normalization. IDF(qi) is the inverse document frequency of term qi, and the calculation formula is usually log((Nn(qi)+0.5) / (n(qi)+0.5)+1), where N is the total number of documents in the document set, n(qi) is the number of documents containing term qi. IDF measures the rarity of a word; the more common the word (such as "the", "a"), the lower its IDF value and the lower its weight.

[0127] Furthermore, this application can sum the scores of all terms ("construction", "worker", "point") to obtain the final BM25_Score of the text document for query Q.

[0128] Finally, this application uses the calculated BM25 score as the keyword matching score, providing a strong basis for subsequent candidate image reordering and screening.

[0129] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the process of refining and semantically verifying each candidate image in the candidate set according to an embodiment of this application; in S140, based on the English text and the English description text corresponding to each candidate image in the candidate set, refining and semantically verifying each candidate image in the candidate set, and outputting the final image set, including:

[0130] S141: Input the English text and the English description text corresponding to each candidate image in the candidate set into the preset scoring model to obtain the correlation score between the English text output by the scoring model and each English description text in the candidate set.

[0131] S142: Based on the correlation scores, the candidate images in the candidate set are finely ranked to obtain the sorted candidate set.

[0132] S143: Perform semantic verification on each candidate image in the sorted candidate set, and filter each candidate image in the sorted candidate set according to the semantic verification results, and output the final image set.

[0133] In this embodiment, when performing fine-grained ranking and semantic verification on each candidate image in the candidate set, this application first inputs the English text and the corresponding English descriptive text for each candidate image in the candidate set into a preset scoring model. This scoring model has been trained on a large amount of data and possesses the ability to accurately assess the relevance between texts. Through the processing of the scoring model, this application can obtain the relevance score between the English text and each English descriptive text in the candidate set; these scores intuitively reflect the degree of matching between the texts.

[0134] After obtaining the relevance scores, this application can refine the ranking of candidate images in the candidate set based on these scores. The candidate images are arranged in descending order of score; a higher score indicates a higher match between the candidate image and the user's query intent, thus obtaining a ranked candidate set. This step helps to prioritize images that best meet the user's needs, improving the quality of search results.

[0135] After ranking, this application can also perform semantic verification on each candidate image in the ranked candidate set. The purpose of semantic verification is to further ensure that the candidate images are semantically consistent with the English text of the user query. Through natural language processing technology and image understanding algorithms, in-depth analysis is performed on the English description text of the candidate images and the user query text to determine whether they are semantically consistent. For example, this application can check whether the objects, scenes, actions, and other elements mentioned in the description text match the query text, as well as whether the emotional atmosphere and artistic style match.

[0136] Based on the semantic verification results, this application can filter each candidate image in the sorted candidate set. Candidate images with semantic mismatch or low matching degree are removed, retaining only those that are semantically highly consistent with the user's query intent. Finally, the output is a refined image set after ranking and semantic verification. The images in this set will more accurately meet the user's needs, providing high-quality image retrieval results and effectively improving the user experience.

[0137] In one embodiment, step S143, which involves performing semantic verification on each candidate image in the sorted candidate set and filtering the candidate images in the sorted candidate set based on the semantic verification results to output the final image set, may include:

[0138] S1431: Using a preset prompt word template, the English description text corresponding to each candidate image in the sorted candidate set is combined with the English text to form multiple questions to be verified.

[0139] S1432: Input each of the questions to be verified into the large language model for reasoning and judgment, and obtain the judgment result output by the large language model corresponding to each of the questions to be verified.

[0140] S1433: Filter each candidate image in the sorted candidate set according to each of the decision results, and output the final image set.

[0141] In this embodiment, to achieve accurate semantic verification of each candidate image in the sorted candidate set, this application innovatively adopts a preset prompt word template approach. Specifically, for each candidate image in the sorted candidate set, this application can cleverly combine its corresponding English description text with the English text input by the user, thereby forming multiple questions to be verified.

[0142] For example, if a user's query text is "a happy child playing in the park", and the English description text of a candidate image is "A little boy with a big smile running on the grass in a sunny park", this application can combine them into the question to be verified using a preset prompt word template: "Is 'A little boy with a big smile running on the grass in a sunny park' describing 'ahappy child playing in the park'?"

[0143] After generating multiple verification questions, this application can input these questions into a large language model for reasoning and judgment. Leveraging its powerful natural language understanding and reasoning capabilities, the large language model can perform in-depth analysis of each verification question and output corresponding judgment results. These judgment results are typically presented in explicit judgment forms, such as "yes" or "no," or provide detailed semantic similarity assessments.

[0144] Finally, this application can rigorously filter each candidate image in the sorted candidate set based on the various judgment results. Candidate images that the large language model determines to be semantically mismatched or with extremely low matching are decisively removed; while candidate images with high semantic matching are retained. Through this series of meticulous operations, a high-quality image set is ultimately output. The images in this set accurately match the user's query intent, providing the user with an excellent image retrieval experience.

[0145] The following describes the multilingual text search and image search device provided in the embodiments of this application. The multilingual text search and image search device described below can be referred to in correspondence with the multilingual text search and image search method described above.

[0146] In one embodiment, such as Figure 4 As shown, Figure 4 This application provides a schematic diagram of the structure of a text search image device supporting multiple languages, as provided in an embodiment of the present application. The present application also provides a text search image device supporting multiple languages, which may include a translation and encoding module 210, a similarity search module 220, an image filtering module 230, and an image reordering module 240, specifically including the following:

[0147] The translation and encoding module 210 is used to receive query text in any language input by the user, translate the query text into English text, and encode the English text into a query vector.

[0148] The similarity search module 220 is used to search in a pre-built vector database for multiple candidate images similar to the query vector, as well as their corresponding English descriptive text and image descriptive vectors.

[0149] The image filtering module 230 is used to reorder and filter each candidate image based on the similarity score between the query vector and each of the image description vectors, and the keyword matching score between the English text and each of the English description texts, to form a candidate set.

[0150] The image reordering module 240 is used to perform fine sorting and semantic verification on each candidate image in the candidate set based on the English text and the English description text corresponding to each candidate image in the candidate set, and output the final image set.

[0151] In the above embodiments, this method receives query text in any language input by the user, translates it into English text using an advanced translation model, and encodes the English text into query vectors, thereby supporting multilingual queries. Simultaneously, the method constructs a vector database containing image description vectors. By calculating the similarity score between the query vector and the image description vector, and the keyword matching score between the English text and the English description text, candidate images are reordered and filtered. This ensures both semantic relevance and keyword hit rate, ultimately outputting an image set highly matching the user's query intent. This process not only improves the accuracy and efficiency of text-to-image search but also greatly expands the user's scope, meeting the information retrieval needs in a multilingual environment.

[0152] In one embodiment, this application also provides a computer-readable storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the multilingual text search image method as described in any of the above embodiments.

[0153] In one embodiment, this application also provides a computer device, including: one or more processors, and memory.

[0154] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the multilingual text search image method as described in any of the above embodiments.

[0155] Indicatively, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 5 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the multilingual text search image method of any of the above embodiments.

[0156] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0157] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0158] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0159] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0160] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for searching images by text in multiple languages, characterized in that, The method comprises: receiving a query text in an arbitrary language input by a user, translating the query text into an English text, and encoding the English text into a query vector; searching a plurality of candidate images and corresponding English description texts and image description vectors similar to the query vector in a pre-constructed vector database; reordering and screening each candidate image according to a similarity score between the query vector and each image description vector and a keyword matching score between the English text and each English description text, to form a candidate set; performing fine arrangement and semantic verification on each candidate image in the candidate set based on the English text and the English description text corresponding to each candidate image in the candidate set, and outputting a final image set.

2. The method for supporting multi-language text search in images according to claim 1, wherein, The translation of the query text into an English text comprises: calling a retrieval-intent-optimized and multi-national-language-supported translation model to translate the query text, and controlling the length of the translated text within a preset length range to obtain an English text output by the translation model.

3. The method for supporting multi-language text search in images according to claim 1, wherein, The construction process of the vector database comprises: obtaining all images in an image library; generating English description texts corresponding to each image based on a structured prompt word guided visual language model, each English description text including at least objects, scenes and actions of the corresponding image; encoding each English description text into an image description vector using a unified English text encoder, and saving each image description vector and related metadata to the vector database.

4. The method for supporting multi-language text search in images according to claim 1, wherein, The reordering and screening of each candidate image to form a candidate set according to a similarity score between the query vector and each image description vector and a keyword matching score between the English text and each English description text comprises: calculating the similarity score between the query vector and each image description vector and the keyword matching score between the English text and each English description text respectively; determining a total score of each candidate image according to each similarity score and each keyword matching score; reordering and screening each candidate image according to the total score of each candidate image to form a candidate set.

5. The method for supporting multi-language text search in images according to claim 4, wherein, The calculation of the keyword matching score between the English text and each English description text comprises: splicing structured fields in each English description text into a complete text document respectively; preprocessing each text document and the English text to obtain a plurality of preprocessed text documents and a preprocessed English text; calculating a BM25 score between the preprocessed English text and each preprocessed text document respectively, and taking the BM25 score as a keyword matching score.

6. The method for supporting multi-language text search in images according to claim 1, wherein, The fine arrangement and semantic verification of each candidate image in the candidate set based on the English text and the English description text corresponding to each candidate image in the candidate set, and the output of a final image set, comprise: input the English text and the English description text corresponding to each candidate image in the candidate set into a preset scoring model to obtain a relevance score between the English text output by the scoring model and each English description text in the candidate set; perform fine sorting on each candidate image in the candidate set according to each relevance score to obtain a sorted candidate set; perform semantic verification on each candidate image in the sorted candidate set, and filter each candidate image in the sorted candidate set according to a semantic verification result to output a final image set.

7. The method for supporting multi-language text search in images according to claim 6, wherein, The performing semantic verification on each candidate image in the sorted candidate set, and filtering each candidate image in the sorted candidate set according to a semantic verification result to output a final image set, includes: combining each English description text corresponding to each candidate image in the sorted candidate set with the English text using a preset prompt word template to form a plurality of to-be-verified questions; inputting each to-be-verified question into a large language model for reasoning and decision-making to obtain a decision result corresponding to each to-be-verified question output by the large language model; filtering each candidate image in the sorted candidate set according to each decision result to output a final image set.

8. A text search image device supporting multi-languages, characterized by, It includes: a translation and encoding module configured to receive a query text in an arbitrary language input by a user, translate the query text into an English text, and encode the English text into a query vector; a similarity search module configured to search for a plurality of candidate images, corresponding English description texts, and image description vectors similar to the query vector in a pre-constructed vector database; an image screening module configured to reorder and screen each candidate image according to a similarity score between the query vector and each image description vector, and a keyword matching score between the English text and each English description text to form a candidate set; an image reordering module configured to perform fine sorting and semantic verification on each candidate image in the candidate set based on the English text and the English description text corresponding to each candidate image in the candidate set to output a final image set.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the text-to-image method supporting multiple languages as claimed in any one of claims 1 to 7.

10. A computer device, comprising: It includes: one or more processors and a memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the text-to-image method supporting multiple languages as claimed in any one of claims 1 to 7.

Images