Multi-mode-based image search function evaluation method, device and equipment and storage medium

Abstract

According to the multi-mode-based image search function evaluation method and device, the equipment and the storage medium, at least one retrieval prompt is obtained according to a search target; in a preset data base library, retrieving to obtain a plurality of retrieval pictures corresponding to each retrieval prompt; calculating to obtain the recall rate of each retrieval prompt; and obtaining a detection result of the search target according to the recall rate of each retrieval prompt. The method has the following technical effects that the workload of a data base library is reduced through more efficient data cleaning and data labeling; automation and intellectualization of the evaluation process are achieved through the artificial intelligence technology, the evaluation efficiency and accuracy are improved, and the problems that in the prior art, the manual judgment workload is large, and judgment is inaccurate are solved.

Description

Technical Field

[0001] This application relates to the fields of deep learning and artificial intelligence, and particularly to a method, device, equipment, and storage medium for evaluating the multi-modal image search function. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, large models have gradually become an important force driving the progress of the artificial intelligence field. With their huge parameter scale, strong generalization ability, and flexible adaptability, large models have achieved remarkable results in multiple fields such as natural language processing, image recognition and generation, and medical diagnosis, showing the dawn of achieving general artificial intelligence. Among them, text-to-image search and image-to-image search, as the two core functions of multi-modal retrieval, respectively refer to searching for relevant images through text descriptions and searching for other similar or relevant images by uploading an image. These two technologies have broad application prospects in multiple fields such as information retrieval, content recommendation, and intelligent customer service.

[0003] A multi-modal large model refers to a deep learning model that can process and understand information from different modalities such as text, images, audio, etc. Such models have extensive applications in fields such as natural language processing and computer vision. Especially in cross-modal retrieval tasks, such as text-to-image search and image-to-image search functions, the text-to-image search technology allows users to search for relevant images by inputting text descriptions. The text descriptions input by users are parsed through natural language processing to extract semantic information, and then feature information is read from the images through computer vision technology. These features need to match the text descriptions. Finally, the text and image features are compared through a cross-modal matching algorithm to find relevant images; for image-to-image search, the technology allows users to search for other similar or relevant images by uploading an image. This requires the model to accurately capture the visual features in the image and perform efficient retrieval in the image library. Deep learning models such as convolutional neural networks are used to extract high-dimensional feature representations from the input image, and the similarity between the input image features and the features of other images in the image library is calculated.

[0004] With the wide application of multi-modal large models in text-to-image search and image-to-image search functions, it becomes particularly important to accurately judge their performance. Traditional evaluation methods often rely on manual annotation and judgment, which is not only time-consuming and laborious but also difficult to ensure the consistency and objectivity of the evaluation. Therefore, developing an automated evaluation method is of great significance for improving evaluation efficiency and reducing evaluation costs. Summary of the Invention

[0005] Embodiments of this application provide a method, device, equipment, and storage medium for evaluating the multi-modal image search function, aiming to solve the problem that in the prior art, only the image feature vectors are used for matching to obtain the retrieved images, and whether the retrieved images are accurate still needs to be evaluated manually for the matching results, resulting in a cumbersome evaluation process and a large workload.

[0006] In a first aspect, an embodiment of the present application provides a method for evaluating a multi-modal image search function, and the method includes:

[0007] Obtain a search target, and based on the search target, obtain at least one retrieval prompt; wherein, the format of the search target is in picture format or text format, and the format of the retrieval prompt is in picture format or text format;

[0008] Based on at least one retrieval prompt, retrieve a number of retrieval images corresponding to each retrieval prompt in a preset data database; wherein, the data database stores a plurality of stored images obtained through data reflux or crawling, and each retrieval image is one of the plurality of stored images;

[0009] Based on each retrieval prompt, obtain the picture name of each retrieval image, and based on at least one retrieval prompt and the picture name of each retrieval image corresponding to each retrieval prompt, calculate the recall rate of each retrieval prompt;

[0010] Based on the recall rate of each retrieval prompt, obtain the detection result of the search target.

[0011] Optionally, in the method as described above, the search target includes at least one feature, and the target feature is any one of the at least one feature;

[0012] For the target feature, obtaining at least one retrieval prompt based on the search target includes:

[0013] Identify the target feature, and based on the target feature, generate a plurality of keywords; wherein, the plurality of keywords are jointly used to describe the target feature;

[0014] Construct sentences with the plurality of keywords to obtain the retrieval prompt corresponding to the target feature; wherein, at least one retrieval prompt corresponds one by one to at least one feature.

[0015] Optionally, in the method as described above, when the format of the search target is in text format, identifying the target feature and generating a plurality of keywords based on the target feature includes:

[0016] Through semantic recognition, screen out the text where the target feature is located from the search target;

[0017] Through keyword extraction, extract a plurality of keywords from the text where the target feature is located.

[0018] Optionally, in the method as described above, when the format of the search target is in picture format, identifying the target feature and generating a plurality of keywords based on the target feature includes:

[0019] Through detection box annotation, mark the picture where the target feature is located from the search target;

[0020] Through image feature recognition, multiple keywords are recognized from the picture where the target feature is located.

[0021] Optionally, for the method as described above, according to the recall rate of each retrieval prompt, the detection result of the search target is obtained, including:

[0022] Calculate the intersection over union (IoU) of each retrieval prompt;

[0023] According to the recall rate and intersection over union of each retrieval prompt, the calibration result of the search target is obtained;

[0024] For the retrieval prompt corresponding to the target feature, calculate the intersection over union of each retrieval prompt, including:

[0025] Calculate the overlap degree between the picture where the target feature is located and each retrieval picture corresponding to the target feature;

[0026] According to at least one overlap degree of the target feature, the intersection over union of the retrieval prompt corresponding to the target feature is obtained.

[0027] Optionally, for the method as described above, after retrieving a number of retrieval pictures corresponding to each retrieval prompt from the preset data database according to at least one retrieval prompt, the method further includes:

[0028] Obtain the retrieval difficulty level of each retrieval prompt;

[0029] Create a retrieval table of the search target according to at least one retrieval prompt, and the multiple keywords, a number of retrieval pictures and retrieval difficulty level corresponding to each retrieval prompt;

[0030] Push the retrieval table of the search target;

[0031] For the retrieval prompt corresponding to the target feature, obtain the retrieval difficulty level of each retrieval prompt, including:

[0032] According to the number of multiple keywords corresponding to the target feature, obtain the retrieval difficulty level of the retrieval prompt corresponding to the target feature; wherein, the retrieval difficulty level is positively correlated with the number of corresponding multiple keywords.

[0033] Optionally, for the method as described above, after creating a retrieval table of the search target according to at least one retrieval prompt, and the multiple keywords, a number of retrieval pictures and retrieval difficulty level corresponding to each retrieval prompt, the method further includes:

[0034] Clear the retrieval prompts with zero number of retrieval pictures from at least one retrieval prompt in the retrieval table;

[0035] Generate an annotation document based on the retrieved form after clearing; wherein, the annotation document is used to indicate the retrieval result of the search target

[0036] In a second aspect, an embodiment of the present application provides a multi-modal image search function evaluation device, which includes:

[0037] An acquisition module, configured to acquire a search target and obtain at least one retrieval prompt according to the search target; wherein, the format of the search target is in picture format or text format, and the format of the retrieval prompt is in picture format or text format;

[0038] A retrieval module, configured to retrieve a plurality of retrieval pictures corresponding to each retrieval prompt in a preset data database according to at least one retrieval prompt; wherein, the data database stores a plurality of stored pictures obtained through data return or crawling, and each retrieval picture is one of the plurality of stored pictures;

[0039] A naming module, configured to obtain the picture name of each retrieval picture according to each retrieval prompt, and calculate the recall rate of each retrieval prompt according to at least one retrieval prompt and the picture name of each retrieval picture corresponding to each retrieval prompt;

[0040] A calculation module, configured to obtain the detection result of the search target according to the recall rate of each retrieval prompt.

[0041] Optionally, for the device as described above, the search target includes at least one feature, and the target feature is any one of the at least one feature;

[0042] For the target feature, the retrieval module includes:

[0043] An identification module, configured to identify the target feature and generate a plurality of keywords according to the target feature; wherein, the plurality of keywords are jointly used to describe the target feature;

[0044] A sentence-making module, configured to make sentences with the plurality of keywords to obtain a retrieval prompt corresponding to the target feature; wherein, at least one retrieval prompt corresponds one by one to at least one feature.

[0045] Optionally, for the device as described above, when the format of the search target is in text format, the identification module includes:

[0046] A semantic recognition module, configured to screen out the text where the target feature is located from the search target through semantic recognition;

[0047] A keyword extraction module, configured to extract a plurality of keywords from the text where the target feature is located through keyword extraction.

[0048] Optionally, for the device as described above, when the format of the search target is in picture format, the identification module includes:

[0049] A labeling module, used to label the picture where the target feature is located from the search target through detection box labeling;

[0050] An image recognition module, used to recognize multiple keywords from the picture where the target feature is located through image feature recognition.

[0051] Optionally, for the device as described above, the calculation module includes:

[0052] A first calculation module, used to calculate the intersection-over-union ratio of each retrieval prompt;

[0053] A detection module, used to obtain the detection result of the search target according to the recall rate and intersection-over-union ratio of each retrieval prompt;

[0054] For the retrieval prompt corresponding to the target feature, the first calculation module includes:

[0055] A second calculation module, used to calculate the overlap degree between the picture where the target feature is located and each retrieval picture corresponding to the target feature;

[0056] An overlap module, used to obtain the intersection-over-union ratio of the retrieval prompt corresponding to the target feature according to at least one overlap degree of the target feature.

[0057] Optionally, for the device as described above, based on the multi-modal image search function evaluation device, the device further includes:

[0058] A first acquisition module, used to acquire the retrieval difficulty level of each retrieval prompt;

[0059] A table module, used to create a retrieval table of the search target according to at least one retrieval prompt, and the multiple keywords, several retrieval pictures and retrieval difficulty levels corresponding to each retrieval prompt;

[0060] A push module, used to push the retrieval table of the search target;

[0061] For the retrieval prompt corresponding to the target feature, the table module includes:

[0062] A level module, used to obtain the retrieval difficulty level of the retrieval prompt corresponding to the target feature according to the number of multiple keywords corresponding to the target feature; wherein, the retrieval difficulty level is positively correlated with the number of corresponding multiple keywords.

[0063] Optionally, for the device as described above, based on the multi-modal image search function evaluation device, the device further includes:

[0064] A clearing module, used to clear the retrieval prompts with zero number of retrieval pictures from at least one retrieval prompt in the retrieval table;

[0065] A document module for generating an annotated document according to the retrieved form after cleaning; wherein, the annotated document is used to indicate the retrieval result of the search target.

[0066] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory and a processor;

[0067] The memory stores computer-executable instructions;

[0068] The processor executes the computer-executable instructions stored in the memory to implement the multi-modal image search function evaluation method according to any one of the first aspects.

[0069] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement a multi-modal image search function evaluation method according to any one of the first aspects.

[0070] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, it is used to implement a multi-modal image search function evaluation method according to any one of the first aspects.

[0071] A multi-modal image search function evaluation method, device, device and storage medium provided by an embodiment of the present application, obtain a search target, and obtain at least one retrieval prompt according to the search target; wherein, the format of the search target is a picture format or a text format, and the format of the retrieval prompt is a picture format or a text format; according to at least one retrieval prompt, in a preset data database, retrieve a plurality of retrieval pictures corresponding to each retrieval prompt; wherein, the data database stores a plurality of stored pictures obtained through data reflux or crawling, and each retrieval picture is one of the plurality of stored pictures; according to each retrieval prompt, obtain the picture name of each retrieval picture, and according to at least one retrieval prompt, and the picture name of each retrieval picture corresponding to each retrieval prompt, calculate the recall rate of each retrieval prompt; according to the recall rate of each retrieval prompt, obtain the detection result of the search target. The following technical effects are achieved: according to at least one retrieval prompt, in a preset data database, retrieving a plurality of retrieval pictures corresponding to each retrieval prompt reduces the workload of the data database through more efficient data cleaning and data annotation; by calculating the recall rate of the retrieval prompt, a comprehensive retrieval evaluation standard is available to solve the problem of incomplete evaluation in the prior art; using artificial intelligence technology to realize the automation and intelligence of the evaluation process, improving the efficiency and accuracy of the evaluation to solve the problem of large workload and inaccurate judgment in manual judgment in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0073] Figure 1 Schematic flowchart of a method for evaluating a multi-modal image search function provided by an embodiment of the present invention Figure 1 ;

[0074] Figure 2 Schematic flowchart of a method for evaluating a multi-modal image search function provided by an embodiment of the present invention Figure 2 ;

[0075] Figure 3 Schematic flowchart of a method for evaluating a multi-modal image search function provided by an embodiment of this application Figure 3 ;

[0076] Figure 4 Schematic flowchart of a method for evaluating a multi-modal image search function provided by an embodiment of this application Figure 4 ;

[0077] Figure 5 Schematic flowchart of a method for evaluating a multi-modal image search function provided by an embodiment of this application Figure 5 ;

[0078] Figure 6 A device for evaluating a multi-modal image search function provided by an embodiment of this application;

[0079] Figure 7 An electronic device provided by an embodiment of this application.

[0080] Through the above accompanying drawings, specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments.

[0081] Reference numerals:

[0082] 610 - Processor; 620 - Memory; 630 - Communication component; 640 - Bus; 710 - Acquisition module; 720 - Retrieval module; 730 - Naming module; 740 - Calculation module. Detailed implementation manners

[0083] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of systems and methods consistent with some aspects of the present application as detailed in the appended claims.

[0084] The technical solutions of the present invention will be described in detail below with specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0085] In the embodiments of the present invention, terms such as "first" and "second" are used to distinguish the same or similar items with basically the same functions and roles. Those skilled in the art can understand that the terms "first", "second", etc. do not limit the quantity and execution order, and the terms "first", "second", etc. do not necessarily limit being different. It should be noted that in the embodiments of the present invention, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner. In the embodiments of the present invention, "at least one" means one or more, and "a plurality" means two or more.

[0086] It should be noted that "when... " in the embodiments of the present invention can be at the instant when a certain situation occurs, or within a period of time after a certain situation occurs. The embodiments of the present invention do not make specific limitations on this. In addition, a multi-modal image search function evaluation method provided by the embodiments of the present invention is only an example, and the multi-modal image search function evaluation method may also include more or less content.

[0087] To facilitate a clear description of the technical solutions of the embodiments of the present invention, the following briefly introduces some terms and technologies involved in the embodiments of the present invention:

[0088] Recall rate: Also known as sensitivity or true positive rate, it is an important indicator for evaluating the performance of a classification model. The recall rate measures the proportion of positive examples that the model can correctly identify among all actual positive examples.

[0089] Text-based image search: It means that when a user inputs a text description, a search engine or related system can retrieve in a massive image database according to these descriptions and return the images most similar to the text content.

[0090] Image search: It relies on image recognition technology to search for similar pictures by extracting and analyzing visual features such as the color, shape, and texture of images.

[0091] To clearly understand the technical solution of this application, first, a detailed introduction to the prior art solution will be given.

[0092] In the prior art, text-to-image search performs natural language processing on the input text description, extracts core information such as keywords and themes, converts them into vector representations, extracts features from each picture in the image database to generate feature vectors of the images, matches the text feature vectors with the image feature vectors, and finds the image most similar to the text description by calculating the similarity between the vectors; image search extracts features from the uploaded picture and a large number of pictures stored in the database, performs effective mathematical representations on the extracted features, converts them into a form that can be calculated and compared, stores the represented features in a dedicated database for quick retrieval, sorts the images in the database according to the similarity values, and returns the top several images with higher similarity as the final search results to be presented to the user. In the prior art, sorting according to the degree of similarity has the problem that the search results are inaccurate and need to be evaluated manually for the matching results, and the evaluation process is cumbersome and has a large workload. To solve this problem, this application proposes a multi-modal image search function evaluation method.

[0093] Figure 1 The flowchart of a multi-modal image search function evaluation method provided by an embodiment of the present invention Figure 1 As Figure 1 shown, a multi-modal image search function evaluation method provided by an embodiment of the present invention includes the following steps:

[0094] S101. Obtain a search target and obtain at least one retrieval hint according to the search target.

[0095] Specifically, the format of the search target is in picture format or text format, and the format of the retrieval hint is in picture format or text format; when the search target is a picture, image recognition technology such as object recognition and scene recognition is used to analyze the content in the picture, identify the key elements in the picture such as people, places, objects, text, etc., and use the semantic large model to construct descriptive sentences for the identified key elements; when the search target is text, read the text content, use text analysis tools such as natural language processing tools and text mining tools to deeply analyze the text content, extract information such as keywords and phrases related to the theme or core concept, and construct a retrieval hint corresponding to the text according to the analysis results.

[0096] S102. According to at least one retrieval hint, retrieve several retrieval pictures corresponding to each retrieval hint in the pre-set data repository.

[0097] Specifically, the data database stores multiple stored images obtained through data return or crawling, and each retrieved image is one of the multiple stored images.

[0098] The data database can be a local image database or an image resource stored in the cloud. To collect the all-things dataset, pictures can be intercepted from the cameras put into use every 3 hours for data return, or any image data can be crawled from the Internet, and the dataset can be stored in the database to establish a search database. Ensure that the data database contains a large number of images, and these images have been properly classified, labeled, or indexed for quick retrieval. During specific evaluation, datasets for security scenarios, industrial scenarios, cultural and tourism scenarios, etc. can be established according to the actual situation.

[0099] Parse the input retrieval prompt to extract keywords, phrases, or concepts. Here, if the retrieval prompt contains multiple keywords, the logical relationships between them such as "and", "or", "not", etc. need to be considered and flexibly applied according to the theme or core concept represented by the retrieval prompt. According to the retrieval prompt, construct a query statement or query vector applicable to the data database, submit the retrieval prompt to the retrieval engine of the data database, and the retrieval engine searches for matching images in the data database according to the retrieval prompt and returns a list of images matching the query statement. Depending on the complexity of the retrieval prompt and the size of the data database, the number of returned images may vary. Retrieve several retrieved images corresponding to each retrieval prompt in the pre-set data database according to at least one retrieval prompt, and display these images to the user in an appropriate manner.

[0100] S103. Obtain the image name of each retrieved image according to each retrieval prompt, and calculate the recall rate of each retrieval prompt according to at least one retrieval prompt and the image name of each retrieved image corresponding to each retrieval prompt.

[0101] Specifically, perform a retrieval in the data database using the retrieval prompt to obtain a list of image names of the retrieved images corresponding to each retrieval prompt. For each retrieval prompt, compare the retrieval result with a pre-defined set of relevant images to determine the number of actually retrieved relevant images. For each retrieval prompt, there should be a pre-defined set of relevant images, and these images should be the results returned by the retrieval system. The recall rate refers to the ratio of the number of actually retrieved relevant images to the number of all relevant images that should be retrieved for a certain retrieval prompt. The consensus is as follows: Recall rate = Number of actually retrieved relevant images / Number of all relevant images that should be retrieved.

[0102] S104. Obtain the detection result of the search target according to the recall rate of each retrieval prompt.

[0103] Specifically, check whether the name of the retrieved image hits the output result image. If it hits, calculate the intersection over union (IoU) of the detection box and the annotation box output in the detection result image. If it is greater than or equal to 0.5, record a positive sample. If it does not hit or hits but the calculated result of the IoU of the detection box and the annotation box is less than 0.5, record it as a negative sample. The number of detected objects in the images that are in the search results but not in the retrieved images is recorded as all samples. In the object detection task, the IoU of the detection box and the annotation box is usually used to judge the positioning accuracy of the detection box. If the value of the IoU of the detection box and the annotation box is greater than a preset threshold such as 0.5 or 0.75, it is considered that the detection box is correct.

[0104] A multi-modal image search function evaluation method according to an embodiment of the present invention obtains a search target and obtains at least one retrieval hint according to the search target. Among them, the format of the search target is in image format or text format, and the format of the retrieval hint is in image format or text format. According to at least one retrieval hint, in a preset data database, retrieve a plurality of retrieval images corresponding to each retrieval hint. Among them, the data database stores a plurality of stored images obtained through data return or crawling, and each retrieval image is one of the plurality of stored images. According to each retrieval hint, obtain the image name of each retrieval image, and according to at least one retrieval hint and the image name of each retrieval image corresponding to each retrieval hint, calculate the recall rate of each retrieval hint. According to the recall rate of each retrieval hint, obtain the detection result of the search target. The following technical effects are achieved: According to at least one retrieval hint, in a preset data database, retrieve a plurality of retrieval images corresponding to each retrieval hint, reduce the workload of the data database through more efficient data cleaning and data annotation; have a comprehensive retrieval evaluation standard by calculating the recall rate of the retrieval hint, and solve the problem of incomplete evaluation in the prior art; use artificial intelligence technology to realize the automation and intelligence of the evaluation process, improve the efficiency and accuracy of the evaluation, and solve the problems of large workload and inaccurate judgment in the prior art by manual judgment in the prior art.

[0105] Figure 2 Schematic diagram of the process of a multi-modal image search function evaluation method provided by an embodiment of the present invention Figure 2 Then, a multi-modal image search function evaluation method provided in this embodiment includes the following steps:

[0106] S201. Identify the target features and generate multiple keywords according to the target features.

[0107] Specifically, the search target includes at least one feature, and the target feature is any one of the at least one feature.

[0108] Among them, the multiple keywords are jointly used to describe the target features.

[0109] In image search by image, the target features mainly refer to the key information in the input image, which will be used to search for similar images in the database. These features may include, but are not limited to: object type, object attributes, scene information, background information, specific targets, etc. After identifying the target features, these features need to be converted into keywords that can be used for searching. For example: object type: female person; object attributes: wearing a red long dress, generating keywords: female, red long dress. Expand the keywords: red dress, long dress female.

[0110] S202. Make sentences with multiple keywords to obtain the retrieval prompts corresponding to the target features.

[0111] Specifically, at least one retrieval prompt corresponds one by one to at least one feature.

[0112] According to the annotation information, determine the keywords for making sentences. For example: long sleeves, middle-aged, male, road. Use these keywords to make sentences to form retrieval prompts for image search by image. For example: "A middle-aged man wearing a long-sleeved coat stands on the road."

[0113] S203. Obtain the retrieval difficulty level of each retrieval prompt.

[0114] Specifically, use the annotated attributes to make sentences to form search prompts. According to the annotated attribute information, such as the number of keywords, to distinguish the difficulty level of the cases. For example, 1 - 2 keywords are simple-difficulty samples, 3 - 6 keywords are medium-difficulty samples, and more than 6 keywords are difficult samples.

[0115] S204. Create a retrieval table for the search target according to at least one retrieval prompt, as well as the multiple keywords, several retrieval images and retrieval difficulty levels corresponding to each retrieval prompt.

[0116] Specifically, each retrieval prompt represents a specific search direction or theme. The multiple keywords related to each retrieval prompt will be used in the search engine to more precisely locate relevant images. The number of retrieval images prepared for each retrieval prompt, and these images will be used as references or examples for the search results. According to the complexity of the search prompt and the breadth of the keywords, assign a difficulty level to each retrieval prompt, and provide the image link corresponding to each retrieval prompt. These images will be used in the actual search process to help users more intuitively understand the search target.

[0117] S205. Push the retrieval table of the search target.

[0118] Specifically, by constructing a retrieval table, key information such as the features and similarity of the search target is sorted out and displayed, so that users can quickly locate the required target pictures. Users can view information such as picture names, similarity, and relevant tags in the retrieval table to quickly locate the required target pictures.

[0119] S206. Obtain the retrieval difficulty level of the retrieval prompt corresponding to the target feature according to the number of multiple keywords corresponding to the target feature.

[0120] Specifically, among them, the retrieval difficulty level is positively correlated with the number of corresponding multiple keywords.

[0121] According to the number of keywords, the system determines the retrieval difficulty level. 1 - 2 keywords are simple difficulty samples, 3 - 6 keywords are medium difficulty samples, and more than 6 keywords are difficult samples. The more keywords there are, the greater the retrieval difficulty, because more features need to be precisely matched.

[0122] The technical effect of this solution in this embodiment is: according to at least one retrieval prompt, in the preset data base, several retrieval pictures corresponding to each retrieval prompt are retrieved, and the workload of the data base is reduced through more efficient data cleaning and data annotation.

[0123] Figure 3 It is a schematic flow of a method for evaluating a multi-modal image search function provided by an embodiment of the present application. Figure 3 . As Figure 3 shown, when the format of the search target is in text format, a method for evaluating a multi-modal image search function provided by this embodiment includes the following steps:

[0124] S301. Through semantic recognition, screen out the text where the target feature is located from the search target.

[0125] Specifically, use natural language processing technology to parse the query input by the user, understand its semantic content, and screen out the text related to the target feature from the text descriptions in the database according to the semantic content of the user query.

[0126] S302. Through keyword extraction, extract multiple keywords from the text where the target feature is located.

[0127] Specifically, use a Chinese word segmentation tool to perform word segmentation on the text to obtain a sequence of words or phrases, perform part-of-speech tagging on the segmented result for subsequent processing, and extract a set of keywords from each document. These keywords can better represent the theme and content of the document.

[0128] A method for evaluating the multi-modal image search function according to an embodiment of the present invention can, through one annotation, solve the problem of increasing the workload of the base library data volume and retrieval prompts, automatically clean the data set that meets the evaluation requirements, improve the efficiency, liberate the manpower, and the development direction of the text-to-image search function is to achieve control by text.

[0129] Figure 4 The flowchart of a method for evaluating the multi-modal image search function provided by an embodiment of the present application Figure 4 . As Figure 4 shown, when the format of the search target is a picture format, a method for evaluating the multi-modal image search function provided by this embodiment includes the following steps:

[0130] S401. Through detection box annotation, mark out the picture where the target feature is located from the search target.

[0131] Specifically, the system preprocesses the query picture, including size adjustment, color correction, etc., to ensure that the image quality is suitable for subsequent feature extraction and matching, and sets a detection box to mark out the picture where the target feature is located from the search target.

[0132] S402. Through image feature recognition, identify multiple keywords from the picture where the target feature is located.

[0133] Specifically, preprocess the input picture, including adjusting the image size, removing noise, enhancing the contrast, etc., to improve the accuracy of subsequent feature extraction, and use a deep learning model to extract features from the preprocessed image. These features can be local features or global features of the image.

[0134] S403. Calculate the intersection over union (IoU) of each retrieval prompt.

[0135] Specifically, in the image search by image technology, calculate the intersection over union (IoU) between each retrieval prompt and the original query image, which is usually used to judge the overlapping degree of two image regions.

[0136] S404. Obtain the detection result of the search target according to the recall rate and intersection over union (IoU) of each retrieval prompt.

[0137] Specifically, the image search by image task is usually to find the image similar to the query image or containing the same target in a large image database according to the query image or retrieval prompt provided by the user, calculate the similarity between the features of the query image and the features of the database images, sort the database images according to the similarity, and select the image with the highest similarity as the search result.

[0138] S405. Calculate the overlap degree between the picture where the target feature is located and each retrieval picture corresponding to the target feature.

[0139] Specifically, a similarity calculation algorithm is used to calculate the similarity between the target image and each image in the retrieval image library, and the results are stored in a similarity matrix.

[0140] S406. Obtain the intersection over union (IoU) of the retrieval hint corresponding to the target feature according to at least one overlap degree of the target feature.

[0141] Specifically, in the application of image search by image, the intersection over union (IoU) of the retrieval hint corresponding to the target feature can be obtained according to at least one overlap degree of the target feature.

[0142] A multi-modal image search function evaluation method according to an embodiment of the present invention solves the problem of incomplete evaluation in the prior art by calculating the recall rate of the retrieval hint and having a comprehensive retrieval evaluation standard.

[0143] Figure 5 is a flowchart of a multi-modal image search function evaluation method provided by an embodiment of the present application Figure 5 As Figure 5 shown, a multi-modal image search function evaluation method provided in this embodiment includes the following steps:

[0144] S501. Remove the retrieval hints with zero number of retrieved images from at least one retrieval hint in the retrieval table.

[0145] Specifically, when uploading an image for search, the system returns a retrieval table that lists multiple retrieval hints. Each retrieval hint may have multiple relevant images. The system traverses each retrieval hint in the retrieval table. For each retrieval hint, check the number of images under it. If the number of images under a certain retrieval hint is zero, then remove this retrieval hint from the retrieval table.

[0146] S502. Generate an annotation document according to the cleared retrieval table.

[0147] Specifically, the annotation document is used to indicate the retrieval results of the search target. Using the same deep learning model as when extracting the features of the database images before, extract the features of the image uploaded by the user. According to the sorting results, select the top N images with the highest similarity as the retrieval results, and generate annotation information for each selected similar image.

[0148] A multi-modal image search function evaluation method according to an embodiment of the present invention utilizes artificial intelligence technology to automate and intelligentize the evaluation process, improving the efficiency and accuracy of the evaluation and solving the problems of large workload and inaccurate judgment in manual judgment in the prior art.

[0149] Figure 6A multi-modal image search function evaluation device provided by an embodiment of the present application, the device includes:

[0150] An acquisition module 610, configured to acquire a search target, and obtain at least one retrieval prompt according to the search target; wherein, the format of the search target is a picture format or a text format, and the format of the retrieval prompt is a picture format or a text format;

[0151] A retrieval module 620, configured to retrieve a plurality of retrieval pictures corresponding to each retrieval prompt in a preset data database according to at least one retrieval prompt; wherein, the data database stores a plurality of stored pictures obtained through data return or crawling, and each retrieval picture is one of the plurality of stored pictures;

[0152] A naming module 630, configured to obtain a picture name of each retrieval picture according to each retrieval prompt, and calculate a recall rate of each retrieval prompt according to at least one retrieval prompt and the picture name of each retrieval picture corresponding to each retrieval prompt;

[0153] A calculation module 640, configured to obtain a detection result of the search target according to the recall rate of each retrieval prompt.

[0154] Optionally, for the device as described above, the search target includes at least one feature, and the target feature is any one of the at least one feature;

[0155] For the target feature, the retrieval module 620 includes:

[0156] An identification module, configured to identify the target feature and generate a plurality of keywords according to the target feature; wherein, the plurality of keywords are jointly used to describe the target feature;

[0157] A sentence-making module, configured to make a sentence with the plurality of keywords to obtain a retrieval prompt corresponding to the target feature; wherein, at least one retrieval prompt corresponds to at least one feature one by one.

[0158] Optionally, for the device as described above, when the format of the search target is a text format, the identification module includes:

[0159] A semantic recognition module, configured to screen out the text where the target feature is located from the search target through semantic recognition;

[0160] A keyword extraction module, configured to extract a plurality of keywords from the text where the target feature is located through keyword extraction.

[0161] Optionally, for the device as described above, when the format of the search target is a picture format, the identification module includes:

[0162] The annotation module is used to annotate the picture where the target feature is located from the search targets through bounding box annotation;

[0163] The image recognition module is used to recognize multiple keywords from the picture where the target feature is located through image feature recognition.

[0164] Optionally, for the device as described above, the calculation module 640 includes:

[0165] The first calculation module is used to calculate the intersection over union (IoU) of each retrieval prompt;

[0166] The detection module is used to obtain the detection result of the search target according to the recall rate and intersection over union of each retrieval prompt;

[0167] For the retrieval prompt corresponding to the target feature, the first calculation module includes:

[0168] The second calculation module is used to calculate the overlap degree between the picture where the target feature is located and each retrieval picture corresponding to the target feature;

[0169] The overlap module is used to obtain the intersection over union of the retrieval prompt corresponding to the target feature according to at least one overlap degree of the target feature.

[0170] Optionally, for the device as described above, based on the multi-modal image search function evaluation device, the device further includes:

[0171] The first acquisition module is used to acquire the retrieval difficulty level of each retrieval prompt;

[0172] The table module is used to create a retrieval table of the search target according to at least one retrieval prompt, and the multiple keywords, several retrieval pictures and retrieval difficulty levels corresponding to each retrieval prompt;

[0173] The push module is used to push the retrieval table of the search target;

[0174] For the retrieval prompt corresponding to the target feature, the table module includes:

[0175] The level module is used to obtain the retrieval difficulty level of the retrieval prompt corresponding to the target feature according to the number of multiple keywords corresponding to the target feature; wherein, the retrieval difficulty level is positively correlated with the number of corresponding multiple keywords.

[0176] Optionally, for the device as described above, based on the multi-modal image search function evaluation device, the device further includes:

[0177] The clearing module is used to clear the retrieval prompts with zero number of retrieval pictures from at least one retrieval prompt in the retrieval table;

[0178] A document module for generating an annotated document according to the retrieved form after cleaning; wherein the annotated document is used to indicate the retrieval result of the search target.

[0179] The multi-modal image search function evaluation device provided in this embodiment can execute Figures 1 - 5 the technical solutions of the method embodiments shown, and its implementation principle and technical effects are similar to Figures 1 - 5 the method embodiments shown, and will not be elaborated here one by one.

[0180] Figure 7 An electronic device provided in an embodiment of the present application, the electronic device includes: at least one processor 710 and a memory 720. The electronic device further includes a communication component 730. Wherein, the processor, the memory, and the communication component are connected through a bus 740.

[0181] In a specific implementation process, at least one processor 710 executes computer-executable instructions stored in the memory 720, so that at least one processor 710 is used to implement the multi-modal image search function evaluation method in the above embodiment.

[0182] For the specific implementation process of the processor 710, reference can be made to the above method embodiments, and its implementation principle and technical effects are similar, and will not be elaborated here in this embodiment.

[0183] In the above embodiment, it should be understood that the processor may be a central processing unit (English: Central Processing Unit, abbreviated: CPU), or other general-purpose processors, digital signal processors (English: Digital Signal Processor, abbreviated: DSP), application-specific integrated circuits (English: Application Specific Integrated Circuit, abbreviated: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0184] The memory may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory.

[0185] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0186] The functions implemented for the electronic device and the master control device are described for the solution provided by the embodiments of the present invention. It can be understood that, in order to implement the above functions, the electronic device or the master control device includes the corresponding hardware structure and / or software module for executing each function. Combining the units and algorithm steps of each example described in the embodiments disclosed in the embodiments of the present invention, the embodiments of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiments of the present invention.

[0187] The embodiments of the present application also provide a computer-readable storage medium. Computer-executable instructions are stored in the computer-readable storage medium. When the computer-executable instructions are executed by a processor, they are used to implement a method for evaluating a multi-modal image search function in the above embodiments.

[0188] The above-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0189] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the electronic device or the master control device.

[0190] The embodiment of the present application also provides a computer program product, including a computer program, which is used to implement a method for evaluating the multi-modal image search function in the above embodiment when executed by a processor.

[0191] The computer program is stored in a readable storage medium, and at least one processor can read the computer program from the readable storage medium, and at least one processor executes the computer program to execute the solution provided in any of the above embodiments.

[0192] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disk, or optical disc that can store program codes.

[0193] So far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for evaluating the multi-modal image search function, characterized in that, The method comprises: Acquire a search target, and obtain at least one search prompt based on the search target; wherein the search target is in a picture format or a text format, and the search prompt is in a picture format or a text format; According to the at least one search prompt, a plurality of search images corresponding to each search prompt are retrieved from a preset data base; wherein the data base stores a plurality of stored images obtained by data reflow or crawling, and each search image is one of the plurality of stored images; Obtaining a picture name of each of the search hints according to each of the search hints, and calculating a recall rate of each of the search hints according to the at least one search hint and the picture name of each of the search hints corresponding to the search hints; The detection result of the search target is obtained according to the recall rate of each retrieval prompt.

2. The method according to claim 1, wherein The search target includes at least one feature, the target feature is Any one of the at least one characteristic; For the target feature, obtaining at least one search prompt according to the search target includes: Identifying the target feature and generating a plurality of keywords based on the target feature; wherein the plurality of keywords are collectively used to describe the target feature; The multiple keywords are used to construct sentences to obtain search prompts corresponding to the target feature; wherein the at least one search prompt corresponds to the at least one feature one by one.

3. The method according to claim 2, characterized in that, When the search target is in a text format, identifying the target features and generating a plurality of keywords based on the target features include: Filtering the text containing the target feature from the search target through semantic recognition; Through keyword extraction, the multiple keywords are extracted from the text where the target feature is located.

4. The method according to claim 3, wherein When the format of the search target is a picture format, identifying the target features and generating multiple keywords based on the target features include: Marking the image where the target feature is located from the search target through detection frame marking; The multiple keywords are identified from the picture where the target features are located through image feature recognition.

5. The method according to claim 4, characterized in that Obtaining the detection result of the search target according to the recall rate of each of the retrieval prompts includes: Calculating the intersection-over-union ratio of each of the search prompts; Obtaining a detection result of the search target according to the recall rate and intersection-over-union ratio of each of the retrieval prompts; For the retrieval hint corresponding to the target feature, calculating the intersection-over-union ratio of each retrieval hint includes: Calculating the degree of overlap between the image where the target feature is located and each of the search images corresponding to the target feature; According to at least one degree of overlap of the target feature, an intersection-over-union ratio of search prompts corresponding to the target feature is obtained.

6. The method according to claim 2, characterized in that, After obtaining a plurality of search images corresponding to each search prompt in a preset database according to the at least one search prompt, the method further includes: Obtaining a search difficulty level for each of the search prompts; Create a retrieval table for the search target according to the at least one retrieval prompt, as well as the multiple keywords, several retrieval images, and retrieval difficulty levels respectively corresponding to each of the retrieval prompts; Push the retrieval table of the search target; For the retrieval prompt corresponding to the target feature, obtaining the retrieval difficulty level of each retrieval prompt includes: Obtain the retrieval difficulty level of the retrieval prompt corresponding to the target feature according to the number of multiple keywords corresponding to the target feature; wherein, the retrieval difficulty level is positively correlated with the number of corresponding multiple keywords.

7. The method according to claim 6, characterized in that, After creating the retrieval table of the search target according to the at least one retrieval prompt, as well as the multiple keywords, several retrieval images, and retrieval difficulty levels respectively corresponding to each of the retrieval prompts, the method further includes: Clear the retrieval prompts with zero number of retrieval images from at least one retrieval prompt in the retrieval table; Generate an annotation document according to the cleared retrieval table; wherein, the annotation document is used to indicate the retrieval result of the search target.

8. A multi-modal image search function evaluation device, characterized in that Includes: An acquisition module, configured to acquire a search target and obtain at least one retrieval prompt according to the search target; wherein, the format of the search target is in picture format or text format, and the format of the retrieval prompt is in picture format or text format; A retrieval module, configured to retrieve several retrieval images corresponding to each retrieval prompt in a preset data base according to the at least one retrieval prompt; wherein, the data base stores multiple stored images obtained through data reflux or crawling, and each retrieval image is one of the multiple stored images; A naming module, configured to obtain the picture name of each retrieval image according to each retrieval prompt, and calculate the recall rate of each retrieval prompt according to the at least one retrieval prompt and the picture name of each retrieval image corresponding to each retrieval prompt; A calculation module, configured to obtain the detection result of the search target according to the recall rate of each retrieval prompt.

9. An electronic device, characterized in that, Includes: A memory and a processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the multi-modal image search function evaluation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Computer execution instructions are stored in the computer-readable storage medium, and when the computer execution instructions are executed by a processor, they are used to implement the multi-modal image search function evaluation method according to any one of claims 1 to 6.

Images