Picture retrieval method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a picture retrieval method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a retrieval text corresponding to a picture retrieval request in response to the picture retrieval request, determining a plurality of synonymous texts and a plurality of sub-texts of the retrieval text, the synonymous texts and the retrieval text are compared with first feature data of a plurality of preset first pictures to obtain first similarity data, the sub-texts are compared with second feature data of a plurality of preset second pictures to obtain second similarity data, the second pictures are local pictures of the first pictures, and the first similarity data are obtained by comparing the first similarity data with the second similarity data; and according to the first similarity data and the second similarity data, sorting the plurality of first pictures to obtain a sorting result, and according to the sorting result, generating a retrieval result and feeding back the retrieval result. Through the embodiment of the invention, the intention of the user can be accurately understood, and the accuracy of carrying out picture search on the retrieval text, especially the relatively complex retrieval text, is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for retrieving an image, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of technologies such as artificial intelligence and big data, various types of information retrieval have become more and more convenient and fast.

[0003] Image search by text is a search method that retrieves relevant images by inputting text. Users can find the required images by describing various features of the images they want to retrieve.

[0004] In the related art, a visual macro model is used to take the search text input by the user as a prompt, and the image is searched according to the prompt. However, for more complex search texts, the search accuracy of this solution is low, and it is difficult to understand the user's true intention. Summary of the invention

[0005] In view of the above problems, a method and device for retrieving pictures, an electronic device, and a storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0006] A method for image retrieval, the method comprising:

[0007] In response to the image search request, obtaining a search text corresponding to the image search request;

[0008] Determining a plurality of synonymous texts and a plurality of subtexts of the search text;

[0009] Compare the multiple synonymous texts and the search text with the first feature data of the preset multiple first images to obtain first similarity data;

[0010] Comparing the plurality of subtexts with second feature data of a plurality of preset second pictures to obtain second similarity data; wherein the second pictures are partial pictures of the first pictures;

[0011] sorting the plurality of first images according to the first similarity data and the second similarity data to obtain a sorting result;

[0012] According to the ranking results, retrieval results are generated and fed back.

[0013] Optionally, the first similarity data includes a plurality of first similarity values ​​between each of the first feature data and the plurality of synonymous texts and the search text, the second similarity data includes a plurality of second similarity values ​​between each of the second feature data and the plurality of subtexts, and the sorting of the plurality of first images according to the first similarity data and the second similarity data to obtain a sorting result includes:

[0014] Determining a global similarity index value of each first image according to the first similarity value;

[0015] Determining a local similarity index value of each first image according to the second similarity value;

[0016] Determining a target similarity index value for each first image according to the global similarity index value and the local similarity index value;

[0017] The plurality of first images are sorted according to the target similarity index value to obtain a sorting result.

[0018] Optionally, determining a global similarity index value of each first image according to the first similarity value includes:

[0019] Determine a first average value and a first variance value of a plurality of first similarity values ​​corresponding to each first feature data;

[0020] A global similarity index value of each first picture is determined according to the first average value and the first variance value.

[0021] Optionally, determining the local similarity index value of each first image according to the second similarity value includes:

[0022] Determine a second average value and a second variance value of a plurality of second similarity values ​​corresponding to each second feature data;

[0023] Determine a second similarity index value for each second picture according to the second average value and the second variance value;

[0024] A maximum value among the plurality of second similarity index values ​​corresponding to each first picture is determined as the local similarity index value.

[0025] Optionally, determining a target similarity index value of each first picture according to the global similarity index value and the local similarity index value includes:

[0026] Determine a first weight value corresponding to the global similarity index value, and determine a second weight value corresponding to the local similarity index value;

[0027] According to the first weight and the second weight value, a weighted average value is calculated for the global similarity index value and the local similarity index value to obtain the target similarity index value.

[0028] Optionally, the sorting result is that the plurality of first images are sorted in descending order according to the target similarity values, and the generating and feeding back of the retrieval result according to the sorting result includes:

[0029] In the sorting results, target images whose target similarity values ​​are greater than or equal to a preset threshold are screened, and the retrieval results are generated and fed back based on the target images.

[0030] Optionally, the determining of a plurality of synonymous texts and a plurality of subtexts of the search text comprises:

[0031] Entities and entity relationships of the search text are extracted, and a plurality of synonymous texts and a plurality of subtexts corresponding to the search text are generated according to the entities and the entity relationships.

[0032] A device for image retrieval, comprising:

[0033] A search text acquisition module, used to respond to an image search request and acquire a search text corresponding to the image search request;

[0034] A text processing module, used for determining a plurality of synonymous texts of the search text and determining a plurality of subtexts of the search text;

[0035] A first similarity data determination module, used for comparing the plurality of synonymous texts with the first feature data of the preset plurality of first images, so as to obtain first similarity data;

[0036] A second similarity data determination module compares the plurality of subtexts with second feature data of a plurality of preset second images to obtain second similarity data; wherein the second images are partial images of the first images;

[0037] a sorting result generating module, configured to sort the plurality of first images according to the first similarity data and the second similarity data to obtain a sorting result;

[0038] The retrieval result generation module is used to generate and feed back the retrieval results according to the sorting results.

[0039] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned image retrieval method when executed by the processor.

[0040] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for image retrieval as described above is implemented.

[0041] The embodiment of the present invention has the following advantages: the embodiment of the present invention includes, in response to an image retrieval request, obtaining a retrieval text corresponding to the image retrieval request, determining multiple synonymous texts and multiple subtexts of the retrieval text, comparing the multiple synonymous texts and the retrieval text with the first feature data of multiple preset first images to obtain first similarity data, comparing the multiple subtexts with the second feature data of multiple preset second images to obtain second similarity data, wherein the second image is a partial image of the first image, sorting the multiple first images according to the first similarity data and the second similarity data to obtain a sorting result, generating a retrieval result and feeding back according to the sorting result, the retrieval text can be expressed more finely and richly through the subtexts and synonymous texts, and the user's retrieval intention can be analyzed more comprehensively, and then the first images are sorted according to the first similarity data and the second similarity data to generate a retrieval result, thereby improving the accuracy of image retrieval for the retrieval text, especially the retrieval text with a more complex structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0043] Figure 1 It is a flowchart of a method for retrieving an image provided by an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of generating a sub-prompt and a synonymous prompt provided by an embodiment of the present invention;

[0045] Figure 3 It is a schematic diagram of an algorithm flow for fusion matching of local features and global features provided by an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of the structure of a cross-modal retrieval system provided by an embodiment of the present invention;

[0047] Figure 5 It is a structural block diagram of a picture retrieval device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Cross-modal retrieval uses data from one modality as a query request to retrieve related data from another modality. In an environment where multimodal data is growing exponentially, cross-modal retrieval can make full use of multiple information sources to more accurately retrieve user demand information, which has more important research value. Among them, cross-modal retrieval based on text and image, that is, returning corresponding image results based on the query prompt, has more valuable applications in the security construction of smart cities, intelligent image management, and data mining of downstream tasks.

[0050] In a related technology, zero-shot image-text retrieval is performed based on the migration of a large visual model. However, for complex prompts, directly using a large visual model to extract text-image features and then perform retrieval has relatively poor results. For example, if "a person wearing a yellow top and a backpack" is used as a prompt, the top images returned by the search are "a yellow backpack" and "a person wearing a yellow bag and a top of another color". For another example, if "three people riding a motorcycle" is used as a prompt, there is ambiguity during the search, and images of three people riding three motorcycles will be returned.

[0051] Therefore, for complex prompts, understanding the user's true intention is crucial to the correctness of retrieval results.

[0052] For the analysis of complex prompts, in a related technology, a standard template library is generated by building a basic template, nesting and combining, and using at least one standard template to perform semantic analysis on the sentence. However, there are certain limitations in establishing a template library, which cannot meet the requirements of analyzing any prompt.

[0053] In a related technology, candidate triples are generated through the schema (entity) information of the knowledge graph, and the classification of candidate triples is further implemented based on a general classification algorithm framework, thereby improving the scalability and flexibility of the triple generation method in different complex retrieval scenarios. Because knowledge graphs usually focus more on describing static entities and their relationships, they are relatively lacking in describing event-oriented content such as "three people riding a motorcycle."

[0054] The above-mentioned related technologies have the following problems:

[0055] The method of building templates has many limitations and cannot cover any prompts. When using knowledge graph triples, for prompts about the nature of events or without additional detailed information, triple construction cannot achieve the true analysis of the user's intentions.

[0056] Retrieval results are generally sorted directly based on feature similarity, and lack the fusion processing of retrieval results that can be split into multiple prompts.

[0057] Therefore, the present invention is based on the core idea of ​​splitting the prompt into sub-prompts and generating synonymous prompts, and sorting and outputting the images by comparing the similarities between the synonymous prompts and global image features, and the similarities between the sub-prompts and local image features, so as to achieve accurate understanding and recognition of complex prompts and improve the accuracy of image retrieval.

[0058] Reference Figure 1 , shows a flowchart of a method for image retrieval provided by an embodiment of the present invention, which may specifically include the following steps:

[0059] Step 101, in response to an image search request, obtaining a search text corresponding to the image search request;

[0060] The search text is a text describing the features of the image you want to search for, and can be used as a prompt for searching images.

[0061] For example, the search text may be short sentences such as "a person wearing a yellow shirt and carrying a backpack", "three people riding a motorcycle", etc., as search key sentences; it may also be a combination of words, etc.

[0062] Step 102, determining a plurality of synonymous texts of the search text, and determining a plurality of subtexts of the search text;

[0063] Synonymous texts are texts that have the same meaning as the search text but are expressed and described differently, such as synonymous sentences and synonyms. Through synonymous texts, richer feature information can be obtained to accurately understand user intent.

[0064] For example, for the search text "three people riding a motorcycle", multiple synonymous texts can be obtained, such as "three people riding on a motorcycle together", "three people on the motorcycle", and "three people squeezed on a motorcycle to travel", which enriches and emphasizes the characteristic information of three people riding on a motorcycle together.

[0065] Subtexts are multiple subtexts obtained by splitting the search text.

[0066] For example, for the search text "three people riding a motorcycle", multiple sub-texts such as "three people", "riding", and "a motorcycle" can be obtained.

[0067] In some embodiments of the present invention, the determining of the multiple subtexts of the search text includes:

[0068] Entities and entity relationships of the search text are extracted, and a plurality of subtexts corresponding to the search text are generated according to the entities and the entity relationships.

[0069] In this embodiment, an entity refers to a word or phrase with a specific meaning in a text, usually representing an object, person, place, organization, etc. in the real world; an entity relationship refers to the association or connection between entities. It describes the semantic connection between entities and is usually used to represent the interaction, affiliation, location, etc. between entities.

[0070] By extracting entities and entity relations, corresponding subtexts and synonymous texts can be generated more accurately.

[0071] For example, for the search text "a person wearing a yellow shirt and a backpack", the entities extracted are "shirt", "backpack", and "person", and the entity relationships extracted are "a person wearing a yellow shirt", "a person carrying a backpack", etc. Figure 2 As shown in the figure, the generated subtexts are "a person wearing a yellow shirt" and "a person carrying a backpack", and the generated synonymous texts are "an individual carrying a backpack and wearing a yellow shirt", "a pedestrian wearing yellow clothes and carrying a backpack", "a person wearing yellow clothes and carrying a bag", etc. Through the extraction of entities and entity relationships, the features of yellow clothes and carrying a bag are enriched and emphasized.

[0072] In practical applications, LLM Agent (Large Language Model Agent) can be used to perform multi-task processing on entities and entity relationships of retrieved text to obtain more accurate and rich information. LLMAgent is a program with LLM as the core, which is used to achieve some goals or tasks set by users. LLM obtains feedback information and chooses to use preset or newly created tools (such as functions, etc.) to complete the task in an iterative manner.

[0073] Step 103, comparing the multiple synonymous texts and the search text with the first feature data of the preset multiple first images to obtain first similarity data;

[0074] The first image is the original image. The goal of text-based image search is to retrieve matching images from multiple first images. The first feature data is the feature data corresponding to the first image, which can be data stored in a database in the form of a feature vector. The first similarity data is data that characterizes the similarity between the synonymous text, the search text, and the first image.

[0075] In practical applications, text features may be extracted from synonymous texts and then compared with the first feature data to obtain first similarity data.

[0076] Step 104, comparing the plurality of subtexts with second feature data of a plurality of preset second pictures to obtain second similarity data; wherein the second pictures are partial pictures of the first pictures;

[0077] The second image is a partial image of the first image, which can be obtained by dividing and extracting the pixel area of ​​the original image. For example, for an original image expressing "a person wearing a yellow shirt and carrying a bag", it can include a partial image containing "backpack", a partial image containing "yellow shirt" and other partial images. The second feature data is the feature data corresponding to the second image, which can be data stored in the database in the form of feature vectors. The second similarity data is data representing the degree of similarity between the subtext and the second image.

[0078] In practical applications, text features may be extracted from the subtext and then compared with the second feature data to obtain second similarity data.

[0079] Step 105, sorting the plurality of first images according to the first similarity data and the second similarity data to obtain a sorting result;

[0080] In this embodiment, the first similarity data is obtained by matching the synonymous text of the retrieved text with the first feature data of the first image, which can reflect the global similarity between the retrieved text and the first image, that is, overall similarity matching is performed on the text and the image; the second similarity data is obtained by comparing the subtext of the retrieved text with the second feature data of the local image of the first image, which can reflect the local similarity between the retrieved text and the first image, that is, similarity matching is performed in units of smaller granularity.

[0081] In a specific implementation, the first similarity data and the second similarity data can be used to perform comprehensive analysis and calculation using data analysis, preset algorithms, and other methods to determine the comprehensive matching degree between each first image and the search text, and then sort the first images according to the comprehensive matching degree.

[0082] In some embodiments of the present invention, the first similarity data includes a plurality of first similarity values ​​between each of the first feature data and the plurality of synonymous texts and the search text, the second similarity data includes a plurality of second similarity values ​​between each of the second feature data and the plurality of subtexts, and the sorting of the plurality of first images according to the first similarity data and the second similarity data to obtain a sorting result includes:

[0083] Sub-step 1051, determining a global similarity index value of each first image according to the first similarity value;

[0084] In this embodiment, the first feature data is obtained by performing similarity matching between each first image and all synonymous texts and the original search text, that is, one first image can correspond to multiple first similarity values. For example, for a first image A, the search text is "a person wearing a yellow shirt and a backpack", and similarity calculations are performed on the synonymous texts "an individual carrying a backpack and a yellow shirt", "a pedestrian wearing yellow clothes and carrying a backpack", "a person wearing yellow clothes and carrying a bag" and the original search text "a person wearing a yellow shirt and a backpack", and four first similarity values ​​of similarity a, similarity b, similarity c, and similarity d can be obtained respectively; and so on, similarity calculations are performed on the synonymous texts and the search texts for all first images.

[0085] The second similarity data is obtained by performing similarity matching between each second image and all subtexts, that is, one second image can correspond to multiple second similarity values. For example, for a second image B, similarity calculations are performed on the subtexts "a person wearing a yellow shirt" and "a person carrying a backpack", and two second similarity values ​​of similarity e and similarity f can be obtained respectively; and so on, similarity calculations are performed on the subtexts for all second images.

[0086] The global similarity index value may be a specific numerical value, such as a score, etc., which is used to quantify the overall matching degree between the first image and the search text.

[0087] In some embodiments of the present invention, determining the global similarity index value of each first image according to the first similarity value includes:

[0088] Determine a first average value and a first variance value of a plurality of first similarity values ​​corresponding to each first feature data;

[0089] A global similarity index value of each first picture is determined according to the first average value and the first variance value.

[0090] In this embodiment, the larger the first average value and the smaller the first variance value, the higher the matching degree between the first image and all synonymous texts. That is, based on the principle that "the first image is similar to each synonymous text", the global similarity index value of each first image is determined.

[0091] Exemplarily, the global similarity value may be determined by a preset algorithm, formula, etc. For example, an algorithm is constructed so that the global similarity index value is positively correlated with the first average value and negatively correlated with the first variance value.

[0092] Sub-step 1052, determining a local similarity index value of each first image according to the second similarity value;

[0093] The local similarity index value may be a specific numerical value, such as a score, etc., which is used to quantify the degree of local matching between the first image and the search text.

[0094] In some embodiments of the present invention, determining the local similarity index value of each first image according to the second similarity value includes:

[0095] Determine a second average value and a second variance value of a plurality of second similarity values ​​corresponding to each second feature data;

[0096] Determine a second similarity index value for each second picture according to the second average value and the second variance value;

[0097] A maximum value among the plurality of second similarity index values ​​corresponding to each first picture is determined as the local similarity index value.

[0098] In this embodiment, the second similarity index value is used to quantify the degree of matching between each second image and all sub-texts. The larger the second average value and the smaller the second variance value, the higher the degree of matching between the second image and all sub-texts. That is, the second similarity index value is determined based on the fact that "the second image is similar to each sub-text".

[0099] Exemplarily, the second similarity index value may be determined by a preset algorithm, formula, etc., for example, an algorithm is constructed so that the second similarity index value is positively correlated with the second average value and negatively correlated with the second variance value.

[0100] Then, since the second picture is a partial picture of the first picture, that is, one first picture corresponds to multiple second pictures and multiple second similarity index values, the maximum value is determined from these second similarity index values ​​as the partial similarity index value of the first picture.

[0101] For example, a first image A corresponds to N partial images, namely, the second image B, and also corresponds to N second similarity index values ​​(s1, s2, s3...s n), then by max(s n )Determine the local similarity value of the first image A.

[0102] Sub-step 1053, determining a target similarity index value for each first image according to the global similarity index value and the local similarity index value;

[0103] In some embodiments of the present invention, determining a target similarity index value of each first image according to the global similarity index value and the local similarity index value includes:

[0104] Determine a first weight value corresponding to the global similarity index value, and determine a second weight value corresponding to the local similarity index value;

[0105] According to the first weight and the second weight value, a weighted average value is calculated for the global similarity index value and the local similarity index value to obtain the target similarity index value.

[0106] In this embodiment, the global similarity index value and the local similarity index value are weighted and averaged by the first weight and the second weight value, and the obtained result is the target similarity index value, which can quantitatively reflect the comprehensive matching degree between the retrieved text and each first image, with higher accuracy and reliability.

[0107] Sub-step 1054, sorting the multiple first images according to the target similarity index value to obtain a sorting result.

[0108] Exemplarily, the first pictures may be sorted in descending order according to the target similarity index values ​​to obtain a sorting result.

[0109] Step 106: Generate and feed back search results based on the ranking results.

[0110] In practical applications, the sorting results can be fed back as complete search results. The top ranked images are the first images with the highest comprehensive matching degree with the search text, that is, the image search results that users expect to obtain. The sorting results can also be screened and filtered before being fed back to users.

[0111] In some embodiments of the present invention, the sorting result is that the plurality of first images are sorted in descending order according to the target similarity values, and the generating and feeding back of the retrieval result according to the sorting result includes:

[0112] In the sorting results, target images whose target similarity values ​​are greater than or equal to a preset threshold are screened, and the retrieval results are generated and fed back based on the target images.

[0113] In this embodiment, a target image whose target similarity value is greater than or equal to a preset threshold is an image retrieval result that matches the retrieval text, and the target image is fed back as a retrieval result.

[0114] In practical applications, the target similarity value corresponding to each image can also be included in the search results for user reference.

[0115] In some embodiments of the present invention, Figure 3 As shown, an algorithm flow for fusing and matching global feature index values ​​and local feature index values ​​is also provided, as follows:

[0116] Pre-build a large image feature library, i.e., a database constructed by extracting feature data from the complete original image (large image), and a small image feature library, i.e., a database constructed by extracting feature data from a partial image (small image) of each original image in the large image feature library;

[0117] Then, receiving the search text (prompt_o) input, splitting prompt_o into N sub-prompts: prompt_sub1, prompt_sub2...prompt_subN, and generating M synonymous prompts: prompt_sa1, prompt_sa2...prompt_saN;

[0118] Further, similarity calculation is performed on each small image feature data F_sub1, F_sub2..F_subN in the small image feature library and all sub-prompts respectively, and similarity calculation is performed on each large image feature data F_sa1, F_sa2..F_saM in the large image feature library and all synonymous prompts and the original prompt_o respectively;

[0119] Calculate the mean and variance of the similarities corresponding to the N sub-prompts, and score the feature data of each small image based on the principle of "maximum mean and minimum variance", and select the largest score as the local matching score of the large image;

[0120] Calculate the mean and variance of the similarities between the M synonymous prompts and the original prompt_o, and score each large image feature data based on the principle of "maximum mean and minimum variance" to obtain the global matching score of the large image;

[0121] Then, the local matching scores and the global matching scores are comprehensively calculated, such as weighted averaging, to obtain the comprehensive score of the large image, and the large images are sorted according to the comprehensive scores. The large images with comprehensive scores greater than or equal to a certain threshold are selected and returned as retrieval results.

[0122] In some embodiments of the present invention, Figure 4 As shown, a cross-modal retrieval system is also provided, which mainly includes a prompt parsing module, a feature parsing module, and a feature matching module, as follows:

[0123] The system receives the input prompt, extracts entities and entity relationships from the prompt through the prompt parsing module, and generates a sub-prompt list and a synonym prompt list.

[0124] Input the sub-prompt list and synonymous prompt list into the feature parsing module to extract the text features of the sub-prompt, synonymous prompt, and original prompt;

[0125] The above text features are input into the feature matching module, the text features of the sub-prompt are matched with the small image feature data in the small image library, the synonymous prompt, the original prompt and the large image feature data in the large image library are matched, and the above matching results are analyzed and integrated to generate the retrieval results.

[0126] The embodiment of the present invention has the following advantages: the embodiment of the present invention includes, in response to an image retrieval request, obtaining a retrieval text corresponding to the image retrieval request, determining multiple synonymous texts and multiple subtexts of the retrieval text, comparing the multiple synonymous texts and the retrieval text with the first feature data of multiple preset first images to obtain first similarity data, comparing the multiple subtexts with the second feature data of multiple preset second images to obtain second similarity data, wherein the second image is a partial image of the first image, sorting the multiple first images according to the first similarity data and the second similarity data to obtain a sorting result, generating a retrieval result and feeding back according to the sorting result, the retrieval text can be expressed more finely and richly through the subtexts and synonymous texts, and the user's retrieval intention can be analyzed more comprehensively, and then the first images are sorted according to the first similarity data and the second similarity data to generate a retrieval result, thereby improving the accuracy of image retrieval for the retrieval text, especially the retrieval text with a more complex structure.

[0127] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0128] Reference Figure 5, shows a schematic diagram of the structure of an image retrieval device provided by an embodiment of the present invention, which may specifically include the following modules:

[0129] A search text acquisition module 501 is used to respond to an image search request and acquire a search text corresponding to the image search request;

[0130] A text processing module 502, used to determine multiple synonymous texts of the search text and multiple subtexts of the search text;

[0131] A first similarity data determination module 503, used for comparing the plurality of synonymous texts with the first feature data of the preset plurality of first images to obtain first similarity data;

[0132] A second similarity data determination module 504 compares the plurality of subtexts with second feature data of a plurality of preset second images to obtain second similarity data; wherein the second images are partial images of the first images;

[0133] A sorting result generating module 505, configured to sort the plurality of first images according to the first similarity data and the second similarity data to obtain a sorting result;

[0134] The search result generation module 506 is used to generate and feed back search results based on the ranking results.

[0135] In some embodiments of the present invention, the first similarity data includes a plurality of first similarity values ​​between each of the first feature data and the plurality of synonymous texts and the search text, the second similarity data includes a plurality of second similarity values ​​between each of the second feature data and the plurality of subtexts, and the ranking result generating module 505 includes:

[0136] A global similarity index value determination submodule, used to determine a global similarity index value of each first image according to the first similarity value;

[0137] A local similarity index value determination submodule, used to determine a local similarity index value of each first image according to the second similarity value;

[0138] a target similarity index value determination submodule, configured to determine a target similarity index value of each first image according to the global similarity index value and the local similarity index value;

[0139] The sorting result generating submodule is used to sort the plurality of first images according to the target similarity index value to obtain a sorting result.

[0140] In some embodiments of the present invention, the global similarity index value determination submodule is further used to:

[0141] Determine a first average value and a first variance value of a plurality of first similarity values ​​corresponding to each first feature data;

[0142] A global similarity index value of each first picture is determined according to the first average value and the first variance value.

[0143] In some embodiments of the present invention, the local similarity index value determination submodule is further used to:

[0144] Determine a second average value and a second variance value of a plurality of second similarity values ​​corresponding to each second feature data;

[0145] Determine a second similarity index value for each second picture according to the second average value and the second variance value;

[0146] A maximum value among the plurality of second similarity index values ​​corresponding to each first picture is determined as the local similarity index value.

[0147] In some embodiments of the present invention, the target similarity index value determination submodule is further used to:

[0148] Determine a first weight value corresponding to the global similarity index value, and determine a second weight value corresponding to the local similarity index value;

[0149] According to the first weight and the second weight value, a weighted average value is calculated for the global similarity index value and the local similarity index value to obtain the target similarity index value.

[0150] In some embodiments of the present invention, the sorting result is that the plurality of first images are sorted in descending order according to the target similarity value, and the search result generating module 506 includes:

[0151] In the sorting results, target images whose target similarity values ​​are greater than or equal to a preset threshold are screened, and the retrieval results are generated and fed back based on the target images.

[0152] In some embodiments of the present invention, the text processing module 502 includes:

[0153] The entity and relationship extraction submodule is used to extract the entities and entity relationships of the search text, and generate multiple synonymous texts and multiple subtexts corresponding to the search text according to the entities and the entity relationships.

[0154] Some embodiments of the present invention further provide an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the above-mentioned image retrieval method is implemented.

[0155] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned image retrieval method is implemented.

[0156] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above-mentioned image retrieval method when executed by a processor.

[0157] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0159] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0160] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0161] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0164] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0165] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0166] The above is a detailed introduction to the image retrieval method and device, electronic device, and storage medium provided. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for image retrieval, characterized in that: The method comprises: In response to the image search request, obtaining a search text corresponding to the image search request; Determining a plurality of synonymous texts and a plurality of subtexts of the search text; Compare the multiple synonymous texts and the search text with the first feature data of the preset multiple first images to obtain first similarity data; Comparing the plurality of subtexts with second feature data of a plurality of preset second pictures to obtain second similarity data; wherein the second pictures are partial pictures of the first pictures; sorting the plurality of first images according to the first similarity data and the second similarity data to obtain a sorting result; According to the ranking results, retrieval results are generated and fed back.

2. The method according to claim 1, characterized in that The first similarity data includes a plurality of first similarity values ​​between each of the first feature data and the plurality of synonymous texts and the search text, the second similarity data includes a plurality of second similarity values ​​between each of the second feature data and the plurality of subtexts, and the plurality of first images are sorted according to the first similarity data and the second similarity data to obtain a sorting result, including: Determining a global similarity index value of each first image according to the first similarity value; Determining a local similarity index value of each first image according to the second similarity value; Determining a target similarity index value for each first image according to the global similarity index value and the local similarity index value; The plurality of first images are sorted according to the target similarity index value to obtain a sorting result.

3. The method according to claim 2, characterized in that The determining, according to the first similarity value, a global similarity index value of each first image includes: Determine a first average value and a first variance value of a plurality of first similarity values ​​corresponding to each first feature data; A global similarity index value of each first picture is determined according to the first average value and the first variance value.

4. The method according to claim 2, characterized in that: The determining, according to the second similarity value, a local similarity index value of each first image includes: Determine a second average value and a second variance value of a plurality of second similarity values ​​corresponding to each second feature data; Determine a second similarity index value for each second picture according to the second average value and the second variance value; A maximum value among the plurality of second similarity index values ​​corresponding to each first picture is determined as the local similarity index value.

5. The method according to claim 2, characterized in that: The step of determining a target similarity index value of each first picture according to the global similarity index value and the local similarity index value includes: Determine a first weight value corresponding to the global similarity index value, and determine a second weight value corresponding to the local similarity index value; According to the first weight and the second weight value, a weighted average value is calculated for the global similarity index value and the local similarity index value to obtain the target similarity index value.

6. The method according to claim 2, characterized in that The sorting result is that the plurality of first images are sorted in descending order according to the target similarity value, and the generating and feeding back of the retrieval result according to the sorting result includes: In the sorting results, target images whose target similarity values ​​are greater than or equal to a preset threshold are screened, and the retrieval results are generated and fed back based on the target images.

7. The method according to any one of claims 1 to 6, characterized in that: The step of determining a plurality of synonymous texts and a plurality of subtexts of the search text comprises: Entities and entity relationships of the search text are extracted, and a plurality of synonymous texts and a plurality of subtexts corresponding to the search text are generated according to the entities and the entity relationships.

8. A device for image retrieval, characterized in that: The device comprises: A search text acquisition module, used to respond to an image search request and acquire a search text corresponding to the image search request; A text processing module, used for determining a plurality of synonymous texts of the search text and determining a plurality of subtexts of the search text; A first similarity data determination module, used for comparing the plurality of synonymous texts with the first feature data of the preset plurality of first images, so as to obtain first similarity data; A second similarity data determination module compares the plurality of subtexts with second feature data of a plurality of preset second images to obtain second similarity data; wherein the second images are partial images of the first images; a sorting result generating module, configured to sort the plurality of first images according to the first similarity data and the second similarity data to obtain a sorting result; The retrieval result generation module is used to generate and feed back the retrieval results according to the sorting results.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for image retrieval as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for image retrieval according to any one of claims 1 to 7 is implemented.

Images