Image retrieval method and device, electronic equipment and storage medium

Abstract

The invention provides an image retrieval method and device, electronic equipment and a storage medium. According to the image retrieval method, when retrieval input data of a user is received, the data type of the retrieval input data is determined; then generating a feature vector corresponding to the retrieval input data according to the data type of the retrieval input data; performing similarity matching on the feature vector and the feature vector of each image in a database to obtain a similarity score; and finally, taking the first n images with the highest similarity scores as retrieval results. Therefore, by means of the method, the feature vector corresponding to the retrieval input data can be generated according to the data type of the retrieval input data. According to the image retrieval method and device, similarity matching is carried out on the feature vectors and the feature vectors of all the images in the database to obtain the similarity score, the retrieval result is returned according to the similarity score, and the image retrieval accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of data retrieval technology, and in particular to an image retrieval method, device, electronic device and storage medium. Background Art

[0002] Data retrieval is increasingly used in daily life, including text data retrieval, image data retrieval, and so on.

[0003] However, in the prior art, the effect of image data retrieval depends largely on the pre-trained dataset of the retriever. If data that is very different from the training dataset is encountered during the retrieval stage, the retrieval effect will be relatively poor. Summary of the invention

[0004] In view of this, the present application provides an image retrieval method, device, electronic device and storage medium to solve the above problems.

[0005] To achieve the above objectives, this application provides the following technical solutions:

[0006] The first aspect of the present application discloses an image retrieval method, comprising:

[0007] When receiving the search input data from the user, determining the data type of the search input data; wherein the data type includes text, voice and image;

[0008] Generate a feature vector corresponding to the search input data according to the data type; wherein the feature vector includes a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector;

[0009] Performing similarity matching between the feature vector and the feature vectors of each image in the database to obtain a similarity score;

[0010] The first n images with the highest similarity scores are taken as retrieval results.

[0011] Optionally, in the above method, generating a feature vector corresponding to the search input data according to the data type includes:

[0012] If the data type is text, preprocessing the search input data to obtain processed text;

[0013] The processed text is encoded by a text encoder to obtain a text semantic vector corresponding to the processed text.

[0014] Optionally, in the above method, generating a feature vector corresponding to the search input data according to the data type includes:

[0015] If the data type is speech, converting the search input data into text through a speech-to-text converter to obtain a converted text;

[0016] Performing data preprocessing on the conversion text to obtain a processed conversion text;

[0017] The processed conversion text is encoded by a text encoder to obtain a text semantic vector corresponding to the processed conversion text.

[0018] Optionally, in the above method, generating a feature vector corresponding to the search input data according to the data type includes:

[0019] If the data type is an image, determining a search object corresponding to the search input data; wherein the search object includes a certain object in the image or the entire image;

[0020] Encoding the search object by an image encoder to obtain an image semantic vector corresponding to the search object;

[0021] Extracting line features of the search object through a contour structure encoder to generate a contour structure vector corresponding to the search object;

[0022] Extracting the style features of the search object through an image style encoder to generate a style vector corresponding to the search object;

[0023] Performing face detection on the search object to generate a face ID vector corresponding to the search object;

[0024] The search object is input into a preset model to obtain a text annotation of the search object, and the text annotation is encoded by an encoder to obtain a text semantic vector of the search object.

[0025] Optionally, in the above method, performing similarity matching between the feature vector and the feature vectors of each image in the database to obtain a similarity score includes:

[0026] If the feature vector corresponding to the retrieval input data is a text semantic vector, then obtaining an image semantic vector and a text semantic vector corresponding to each image in the database;

[0027] Calculating a first cosine similarity between the text semantic vector of the retrieval input data and the image semantic vector of each image in the database, and calculating a second cosine similarity between the text semantic vector of the retrieval input data and the text semantic vector of each image in the database;

[0028] Based on the first cosine similarity and the second cosine similarity, a similarity score between the retrieval input data and each image in the database is calculated.

[0029] Optionally, in the above method, performing similarity matching between the feature vector and the feature vectors of each image in the database to obtain a similarity score includes:

[0030] If the feature vector corresponding to the retrieval input data is a text semantic vector, an image semantic vector, a contour structure vector, a style vector, and a face ID vector, then the image semantic vector, the text semantic vector, the contour structure vector, the style vector, and the face ID vector corresponding to each image in the database are obtained;

[0031] Calculating the first cosine similarity between the image semantic vector of the retrieved input data and the image semantic vector of each image in the database;

[0032] Calculating the second cosine similarity between the text semantic vector of the retrieval input data and the text semantic vector of each image in the database;

[0033] Calculating the third cosine similarity between the contour structure vector of the retrieval input data and the contour structure vector of each image in the database;

[0034] Calculating the fourth cosine similarity between the style vector of the retrieved input data and the style vector of each image in the database;

[0035] Calculating the fifth cosine similarity between the face ID vector of the retrieved input data and the face ID vector of each image in the database;

[0036] Based on the first cosine similarity, the second cosine similarity, the third cosine similarity, the fourth cosine similarity and the fifth cosine similarity, a similarity score between the retrieval input data and each image in the database is calculated.

[0037] The second aspect of the present application discloses an image retrieval device, comprising:

[0038] A determination unit, configured to determine the data type of the search input data upon receiving the search input data from the user; wherein the data type includes text, voice and image;

[0039] An encoding unit, used to generate a feature vector corresponding to the search input data according to the data type; wherein the feature vector includes a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector;

[0040] A matching unit, used for performing similarity matching between the feature vector and the feature vectors of each image in the database to obtain a similarity score;

[0041] The screening unit is used to select the first n images with the highest similarity scores as the retrieval results.

[0042] Optionally, in the above device, the encoding unit includes:

[0043] A first preprocessing subunit is used for performing data preprocessing on the search input data to obtain processed text if the data type is text;

[0044] The first encoding subunit is used to encode the processed text through a text encoder to obtain a text semantic vector corresponding to the processed text.

[0045] The third aspect of the present application discloses an electronic device, comprising:

[0046] one or more processors;

[0047] a storage device having one or more programs stored thereon;

[0048] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of the first aspects of the present application.

[0049] A fourth aspect of the present application discloses a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any one of the first aspects of the present application is implemented.

[0050] It can be seen from the above technical scheme that in an image retrieval method provided by the present application, when receiving the user's retrieval input data, the data type of the retrieval input data is determined; wherein the data type includes text, voice and image. Then, according to the data type, a feature vector corresponding to the retrieval input data is generated; wherein the feature vector includes a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector. Then, the feature vector is matched with the feature vectors of each image in the database for similarity to obtain a similarity score. Finally, the top n images with the highest similarity scores are used as retrieval results. It can be seen that, using the method of the present application, feature vectors corresponding to different data types will be generated according to the data type of the retrieval input data, and the feature vectors are subdivided into text semantic features and image semantic features, so that the semantic feature matching is more accurate. The feature vector is used to match the feature vectors of each image in the database for similarity to obtain a similarity score, and the retrieval result is returned according to the similarity score, thereby improving the accuracy of image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0052] Figure 1 A flowchart of an image retrieval method disclosed in an embodiment of the present application;

[0053] Figure 2 This is a flowchart of an implementation of step S102 disclosed in another embodiment of the present application;

[0054] Figure 3 This is an example diagram of determining a search object disclosed in another embodiment of the present application;

[0055] Figure 4 This is a flowchart of an implementation of step S103 disclosed in another embodiment of the present application;

[0056] Figure 5 A schematic diagram of an image retrieval device disclosed in another embodiment of the present application;

[0057] Figure 6 A schematic diagram of an electronic device disclosed in another embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0060] Furthermore, in this document, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0061] The present application embodiment provides a method for image retrieval, such as Figure 1 As shown, specifically including:

[0062] S101. Upon receiving search input data from a user, determine the data type of the search input data; wherein the data type includes text, voice and image.

[0063] It should be noted that when a user performs image retrieval, the image retrieval may be performed by inputting text, voice or image, etc. After receiving the user's retrieval input data, the data type of the retrieval input data is first determined.

[0064] S102. Generate a feature vector corresponding to the search input data according to the data type; wherein the feature vector includes a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector.

[0065] It should be noted that after determining the data type of the retrieval input data, the encoder is used to encode the retrieval input data to obtain the feature vector of the input data. If the retrieval input data is text or speech, the corresponding feature vector is the text semantic vector. If the retrieval input data is an image, the corresponding feature vector is the text semantic vector, image semantic vector, contour structure vector, style vector and face ID vector.

[0066] Optionally, in another embodiment of the present application, an implementation of step S102 may include:

[0067] If the data type is text, data preprocessing is performed on the search input data to obtain processed text.

[0068] The processed text is encoded by a text encoder to obtain a text semantic vector corresponding to the processed text.

[0069] It should be noted that if the data type of the retrieval input data is text, the input text data is preprocessed, for example, the large language model rewrites and expands the input text to eliminate the interference of invalid text in the input text, especially when the user inputs a long text to make it more refined and the theme is clear, and the processed text is obtained. Then, the processed text is encoded by the text encoder of CLIP (Contrastive Language-Image Pre-training Model) to obtain the text semantic vector corresponding to the processed text.

[0070] Optionally, in another embodiment of the present application, an implementation of step S102 may include:

[0071] If the data type is voice, the retrieved input data is converted into text by a voice-to-text converter to obtain a converted text.

[0072] The conversion text is subjected to data preprocessing to obtain a processed conversion text.

[0073] The processed conversion text is encoded by a text encoder to obtain a text semantic vector corresponding to the processed conversion text.

[0074] It should be noted that if the data type of the search input data is speech, the search input data is converted into text by a pre-placed speech-to-text converter to obtain a converted text. The converted text is then preprocessed, for example, a large language model rewrites and expands the converted text to eliminate the interference of invalid text in the converted text to obtain a processed text. The converted text is encoded by a text encoder of CLIP to obtain a text semantic vector corresponding to the converted text.

[0075] Optionally, in another embodiment of the present application, an implementation of step S102 is as follows: Figure 2 As shown, it may include:

[0076] S201. If the data type is an image, determine a search object corresponding to the search input data; wherein the search object includes a certain object in the image or the entire image.

[0077] It should be noted that if the data type of the search input data is an image, an interactive segmentation page is generated to allow the user to select a search object in the image, wherein the search object includes a certain object in the image or the entire image. Figure 3 For the image shown, the retrieval object can be the sofa in the image or the entire image.

[0078] S202: Encode the search object by using an image encoder to obtain an image semantic vector corresponding to the search object.

[0079] It should be noted that after the search object is determined, the search object is encoded by the CLIP image encoder to obtain the image semantic vector corresponding to the search object.

[0080] S203 , extracting line drawing features of the search object through a contour structure encoder, and generating a contour structure vector corresponding to the search object.

[0081] It should be noted that the line drawing features of the search object are extracted through a contour structure encoder, and the contour structure vector corresponding to the search object is encoded and generated, wherein the contour structure vector is used to represent the contour of the search object.

[0082] S204: extracting style features of the search object through an image style encoder, and generating a style vector corresponding to the search object.

[0083] It should be noted that the style features of the search object are extracted through an image style encoder, and a style vector corresponding to the search object is generated by encoding, wherein the style vector is used to represent the style of the search object.

[0084] S205: Perform face detection on the search object to generate a face ID vector corresponding to the search object.

[0085] It should be noted that when generating a face ID vector, face detection is performed on the search object. If a face is detected, the face ID vector is returned, and if no face is detected, the face ID vector is 0.

[0086] S206: Input the search object into a preset model to obtain text annotations of the search object, and encode the text annotations through an encoder to obtain a text semantic vector of the search object.

[0087] It should be noted that the search object is input into a preset model to obtain the text annotation of the search object. The model can be set according to actual conditions, such as a VLLM (Visual Language Large Model) model. Then, the text annotation is encoded by the text encoder of CLIP to obtain the text semantic vector of the search object.

[0088] S103: Perform similarity matching between the feature vector and the feature vectors of each image in the database to obtain a similarity score.

[0089] It should be noted that after obtaining the feature vector corresponding to the search input data, the feature vector is used to perform similarity matching with the feature vectors of each image in the database to obtain a similarity score. The database is used to store images and the feature vector corresponding to each image.

[0090] The construction process of the database is as follows:

[0091] a. Create a dataset: image-text pairing data with labels (descriptive text corresponding to the image), and rewrite, expand and tune the image annotations to generate highly descriptive image annotations.

[0092] b. Train Jina CLIP: Use the rewritten, expanded, and tuned data to train Jina CLIP. This includes:

[0093] 1. Use images and paired short texts that have not been rewritten, expanded, or optimized to initialize the training model so that the model has good retrieval capabilities for short texts;

[0094] 2. Use the rewritten, expanded and tuned synthetic long texts and corresponding images to continue training the model, so that the model has better processing capabilities for long texts, further improving the model's retrieval capabilities;

[0095] 3. Use text containing hard negative samples to further optimize the text encoder and learn to distinguish relevant text from irrelevant text. At the same time, in order to keep the alignment between text and image, continue to train on long text image data so that the model has good retrieval performance for pure text, while the retrieval ability for images remains consistent.

[0096] c. Image and text encoding: Use CLIP's image encoder and text encoder to encode paired images and texts to generate corresponding text semantic vectors and image semantic vectors.

[0097] d. Vector index: Create an index in the vector database and insert the vector representations of the text and images and the corresponding metadata into the vector index. The metadata associated with each vector representation provides additional information about the data, which helps to understand and manage these vectors. Metadata usually includes but is not limited to the following:

[0098] ID: A unique identifier used to distinguish between different vectors.

[0099] Original data source: A link or description to the original data, such as an image file name or text content.

[0100] Feature type: describes the type of feature represented by the vector, such as image features, text embedding, etc.

[0101] Creation Date: The time when the record vector is generated.

[0102] Update history: A record of modifications or updates to a vector.

[0103] Label: A classification or description of a vector, such as an object category or subject.

[0104] Similarity information: Similarity measure with other vectors, for quick retrieval.

[0105] Optionally, in another embodiment of the present application, an implementation of step S103 is as follows: Figure 4 As shown, it may include:

[0106] S401: If the feature vector corresponding to the search input data is a text semantic vector, then obtain an image semantic vector and a text semantic vector corresponding to each image in the database.

[0107] It should be noted that if the feature vector corresponding to the retrieved input data is a text semantic vector, then the image semantic vector and the text semantic vector corresponding to each image in the database are obtained.

[0108] S402, calculating a first cosine similarity between the text semantic vector of the retrieval input data and the image semantic vector of each image in the database, and calculating a second cosine similarity between the text semantic vector of the retrieval input data and the text semantic vector of each image in the database.

[0109] It should be noted that the first cosine similarity between the text semantic vector of the retrieval input data and the image semantic vector of each image in the database is calculated respectively, and the second cosine similarity between the text semantic vector of the retrieval input data and the text semantic vector of each image in the database is calculated. For example, the text semantic vector of the retrieval input data is X, the image semantic vector of an image in the database is I, and the text semantic vector corresponding to this image is T, then:

[0110] The cosine similarity between X and I is cosine_similarity(X, I), which is the first cosine similarity;

[0111] The cosine similarity between X and T is cosine_similarity(X, T), which is the second cosine similarity.

[0112] S403: Calculate a similarity score between the search input data and each image in the database based on the first cosine similarity and the second cosine similarity.

[0113] It should be noted that the first cosine similarity and the second cosine similarity are weighted and summed to calculate the similarity score between the search input data and each image in the database. The calculation formula is as follows:

[0114] final_score = w *cosine_similarity(X, I) + z *cosine_similarity(X, T), where w and z are the preset weight values, and final_score is the similarity score.

[0115] Optionally, in another embodiment of the present application, an implementation of step S103 may include:

[0116] If the feature vector corresponding to the retrieval input data is a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector, then the image semantic vector, text semantic vector, contour structure vector, style vector and face ID vector corresponding to each image in the database are obtained.

[0117] The first cosine similarity between the image semantic vector of the retrieval input data and the image semantic vector of each image in the database is calculated.

[0118] The second cosine similarity between the text semantic vector of the retrieval input data and the text semantic vector of each image in the database is calculated.

[0119] The third cosine similarity between the contour structure vector of the retrieval input data and the contour structure vector of each image in the database is calculated.

[0120] The fourth cosine similarity between the style vector of the retrieved input data and the style vector of each image in the database is calculated.

[0121] The fifth cosine similarity between the face ID vector of the retrieved input data and the face ID vector of each image in the database is calculated.

[0122] Based on the first cosine similarity, the second cosine similarity, the third cosine similarity, the fourth cosine similarity and the fifth cosine similarity, a similarity score between the retrieval input data and each image in the database is calculated.

[0123] It should be noted that if the feature vector corresponding to the retrieval input data is a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector, then the text semantic vector, image semantic vector, contour structure vector, style vector and face ID vector corresponding to each image in the database are obtained. Then the cosine similarity between each vector of the retrieval input data and each vector of each image in the database is calculated respectively. For example, the text semantic vector of the retrieval input data is X, the image semantic vector is Y, the contour structure vector is Z, the style vector is W, the face ID vector is U, the image semantic vector of an image in the database is I, the text semantic vector is T, the style vector is K, the style vector is M, and the face ID vector is N, then:

[0124] The cosine similarity between Y and I is cosine_similarity(Y, I), which is the first cosine similarity;

[0125] The cosine similarity between X and T is cosine_similarity(X, T), which is the second cosine similarity;

[0126] The cosine similarity between Z and K is cosine_similarity(Z, K), which is the third cosine similarity;

[0127] The cosine similarity between W and M is cosine_similarity(W, M), which is the fourth cosine similarity;

[0128] The cosine similarity between U and N is cosine_similarity(U, N), which is the fifth cosine similarity.

[0129] Finally, the first cosine similarity, the second cosine similarity, the third cosine similarity, the fourth cosine similarity and the fifth cosine similarity are weighted and summed to obtain the similarity score between the search input data and each image in the database. The calculation process can be referred to the implementation of the above step S403, which will not be repeated here.

[0130] S104: Take the first n images with the highest similarity scores as retrieval results.

[0131] It should be noted that after calculating the similarity scores between the feature vector of the search input data and each image in the database, the first n images with the highest similarity scores are returned to the user as search results.

[0132] It can be seen from the above embodiments that the method of the present application has the following advantages:

[0133] (1) Multimodal expansion and enhancement: text-to-text (label and image annotation expansion), text-to-image, text-to-video, image-to-image, and voice-to-image;

[0134] (2) Improved search result accuracy: through more dimensional similarity indicators, such as semantic similarity based on CLIP, contour structure similarity based on line drawing features, style similarity based on style feature extraction, and ID similarity based on face ID features;

[0135] (3) Improved image search granularity: Implement object-level image search based on SAM, use SAM to extract targets and fill in backgrounds, remove interference from other targets, and achieve more refined semantic matching;

[0136] (4) Supporting longer text input and being able to expand the input text: Training and enhancing the CLIP encoder to represent fine-grained information;

[0137] (5) The data was expanded, rewritten, and fine-tuned using LLM and VLLM to improve CLIP training, showing diversity in sentence structure and vocabulary while retaining the original key concepts and meanings;

[0138] (6) The semantic features of CLIP are subdivided into text semantic features and image semantic features to make the semantic feature matching more accurate.

[0139] In an image retrieval method provided by an embodiment of the present application, when receiving the user's retrieval input data, the data type of the retrieval input data is determined; wherein the data type includes text, voice and image. Then, according to the data type, a feature vector corresponding to the retrieval input data is generated; wherein the feature vector includes a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector. Then, the feature vector is matched with the feature vectors of each image in the database for similarity to obtain a similarity score. Finally, the top n images with the highest similarity scores are used as retrieval results. It can be seen that using the method of the present application, feature vectors corresponding to different data types will be generated according to the data type of the retrieval input data, and the feature vectors are subdivided into text semantic features and image semantic features, so that the semantic feature matching is more accurate. The feature vector is used to match the feature vectors of each image in the database for similarity to obtain a similarity score, and the retrieval result is returned according to the similarity score, thereby improving the accuracy of image retrieval.

[0140] In another embodiment of the present application, a device for image retrieval is also provided. Figure 5 As shown, specifically including:

[0141] The determination unit 501 is used to determine the data type of the search input data upon receiving the search input data from the user; wherein the data type includes text, voice and image.

[0142] The encoding unit 502 is used to generate a feature vector corresponding to the search input data according to the data type; wherein the feature vector includes a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector.

[0143] The matching unit 503 is used to perform similarity matching between the feature vector and the feature vectors of each image in the database to obtain a similarity score.

[0144] The screening unit 504 is used to select the first n images with the highest similarity scores as the retrieval results.

[0145] In this embodiment, the specific execution process of the determination unit 501, the encoding unit 502, the matching unit 503 and the screening unit 504 can be referred to in the corresponding Figure 1 The content of the method embodiment will not be repeated here.

[0146] In an image retrieval device provided by an embodiment of the present application, when the determination unit 501 receives the user's retrieval input data, it determines the data type of the retrieval input data; wherein the data type includes text, voice and image. Then the encoding unit 502 generates a feature vector corresponding to the retrieval input data according to the data type; wherein the feature vector includes a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector. The matching unit 503 then performs similarity matching on the feature vector with the feature vectors of each image in the database to obtain a similarity score. Finally, the screening unit 504 uses the top n images with the highest similarity scores as the retrieval results. It can be seen that, using the method of the present application, feature vectors corresponding to different data types will be generated according to the data type of the retrieval input data, and the feature vectors are subdivided into text semantic features and image semantic features, so that the semantic feature matching is more accurate. The feature vector is used to perform similarity matching on the feature vectors of each image in the database to obtain a similarity score, and the retrieval result is returned according to the similarity score, thereby improving the accuracy of image retrieval.

[0147] Optionally, in another embodiment of the present application, an implementation of the encoding unit 502 may include:

[0148] A first preprocessing subunit is used for performing data preprocessing on the search input data to obtain processed text if the data type is text;

[0149] The first encoding subunit is used to encode the processed text through a text encoder to obtain a text semantic vector corresponding to the processed text.

[0150] In this embodiment, the specific execution process of the first preprocessing subunit and the first encoding subunit can refer to the corresponding method embodiment content mentioned above, which will not be repeated here.

[0151] Another embodiment of the application also provides an electronic device, such as Figure 6 As shown, specifically including:

[0152] One or more processors 601 .

[0153] The storage device 602 stores one or more programs.

[0154] When one or more programs are executed by one or more processors 601 , the one or more processors 601 implement any one of the methods in the above embodiments.

[0155] Another embodiment of the present application further provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, any one of the methods in the above embodiments is implemented.

[0156] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0157] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0158] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for image retrieval, characterized in that: include: When receiving the search input data from the user, determining the data type of the search input data; wherein the data type includes text, voice and image; Generate a feature vector corresponding to the search input data according to the data type; wherein the feature vector includes a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector; Performing similarity matching between the feature vector and the feature vectors of each image in the database to obtain a similarity score; The first n images with the highest similarity scores are taken as retrieval results.

2. The method according to claim 1, characterized in that The step of generating a feature vector corresponding to the search input data according to the data type includes: If the data type is text, preprocessing the search input data to obtain processed text; The processed text is encoded by a text encoder to obtain a text semantic vector corresponding to the processed text.

3. The method according to claim 1, characterized in that The step of generating a feature vector corresponding to the search input data according to the data type includes: If the data type is speech, converting the search input data into text through a speech-to-text converter to obtain a converted text; Performing data preprocessing on the conversion text to obtain a processed conversion text; The processed conversion text is encoded by a text encoder to obtain a text semantic vector corresponding to the processed conversion text.

4. The method according to claim 1, characterized in that The step of generating a feature vector corresponding to the search input data according to the data type includes: If the data type is an image, determining a search object corresponding to the search input data; wherein the search object includes a certain object in the image or the entire image; Encoding the search object by an image encoder to obtain an image semantic vector corresponding to the search object; Extracting line features of the search object through a contour structure encoder to generate a contour structure vector corresponding to the search object; Extracting the style features of the search object through an image style encoder to generate a style vector corresponding to the search object; Performing face detection on the search object to generate a face ID vector corresponding to the search object; The search object is input into a preset model to obtain a text annotation of the search object, and the text annotation is encoded by an encoder to obtain a text semantic vector of the search object.

5. The method according to claim 1, characterized in that The similarity matching of the feature vector with the feature vectors of each image in the database to obtain a similarity score includes: If the feature vector corresponding to the retrieval input data is a text semantic vector, then obtaining an image semantic vector and a text semantic vector corresponding to each image in the database; Calculating a first cosine similarity between the text semantic vector of the retrieval input data and the image semantic vector of each image in the database, and calculating a second cosine similarity between the text semantic vector of the retrieval input data and the text semantic vector of each image in the database; Based on the first cosine similarity and the second cosine similarity, a similarity score between the retrieval input data and each image in the database is calculated.

6. The method according to claim 1, characterized in that The similarity matching of the feature vector with the feature vectors of each image in the database to obtain a similarity score includes: If the feature vector corresponding to the retrieval input data is a text semantic vector, an image semantic vector, a contour structure vector, a style vector, and a face ID vector, then the image semantic vector, the text semantic vector, the contour structure vector, the style vector, and the face ID vector corresponding to each image in the database are obtained; Calculating the first cosine similarity between the image semantic vector of the retrieved input data and the image semantic vector of each image in the database; Calculating the second cosine similarity between the text semantic vector of the retrieval input data and the text semantic vector of each image in the database; Calculating the third cosine similarity between the contour structure vector of the retrieval input data and the contour structure vector of each image in the database; Calculating the fourth cosine similarity between the style vector of the retrieved input data and the style vector of each image in the database; Calculating the fifth cosine similarity between the face ID vector of the retrieved input data and the face ID vector of each image in the database; Based on the first cosine similarity, the second cosine similarity, the third cosine similarity, the fourth cosine similarity and the fifth cosine similarity, a similarity score between the retrieval input data and each image in the database is calculated.

7. An image retrieval device, characterized in that: include: A determination unit, configured to determine the data type of the search input data upon receiving the search input data from the user; wherein the data type includes text, voice and image; An encoding unit, used to generate a feature vector corresponding to the search input data according to the data type; wherein the feature vector includes a text semantic vector, an image semantic vector, a contour structure vector, a style vector and a face ID vector; A matching unit, used for performing similarity matching between the feature vector and the feature vectors of each image in the database to obtain a similarity score; The screening unit is used to select the first n images with the highest similarity scores as the retrieval results.

8. The device according to claim 7, characterized in that The encoding unit comprises: A first preprocessing subunit is used for performing data preprocessing on the search input data to obtain processed text if the data type is text; The first encoding subunit is used to encode the processed text through a text encoder to obtain a text semantic vector corresponding to the processed text.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 6.

10. A computer storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images