Image searching method and device, electronic equipment and storage medium

Abstract

The invention relates to an image searching method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring description information of a to-be-searched image; extracting key field information in the description information; determining a first alternative image set in an image library based on the key field information; extracting text features in the description information by using a preset text and image association model, and determining a second alternative image set in the image library based on the text features; and based on the first alternative image set and the second alternative image set, determining a target image matched with the description information. Through the method, the accuracy and / or recall rate of image search can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image search technology, and in particular to an image search method and device, an electronic device, and a storage medium. Background Art

[0002] With the development of the Internet and photography technology, image data is being generated and disseminated at an explosive rate. For example, users continuously accumulate various image data (photographing, transmitting, editing, etc.) when using their devices. Image search technology is committed to enabling users to quickly and accurately find and access the required images in various image data. The current mainstream image search methods still have the problem of limited support for search methods and low accuracy. Summary of the invention

[0003] The present disclosure provides an image search method and device, an electronic device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image search method, comprising:

[0005] Get the description information of the image to be searched;

[0006] Extract key field information from the description information;

[0007] Determining a first candidate image set in an image library based on the key field information;

[0008] Extracting text features in the description information using a preset text-image association model, and determining a second candidate image set in the image library based on the text features;

[0009] Based on the first candidate image set and the second candidate image set, a target image matching the description information is determined.

[0010] In some embodiments, each image in the image library is associated with image metadata, and / or each image is associated with an optical character recognition result;

[0011] The determining a first candidate image set in an image library based on the key field information includes:

[0012] Matching the key field information with the image metadata of each image in the image library, and selecting a first matching set from the image library; and / or,

[0013] Matching the key field information with an optical character recognition result of each image in the image library, and selecting a second matching set from the image library;

[0014] The first candidate image set is determined according to the first matching set and / or the second matching set.

[0015] In some embodiments, the key field information includes attribute identification information and / or content identification information; wherein the attribute identification information includes at least one of the following: time information, location information.

[0016] In some embodiments, the extracting text features in the description information using a preset text-image association model, and determining a second candidate image set in the image library based on the text features, includes:

[0017] Extracting text features in the description information by using the text-image association model, and matching the text features with image features of each image in the image library to obtain a matching result between the description information and each image; wherein the image features of each image in the image library are obtained based on the text-image association model;

[0018] The second candidate image set is determined according to the matching result between the description information and each image.

[0019] In some embodiments, each image in the first candidate image set and the second candidate image set is associated with a matching degree, and determining a target image matching the description information based on the first candidate image set and the second candidate image set includes:

[0020] For each identical image in the first candidate image set and the second candidate image set, determining a target matching degree of the image based on matching degrees of the image in the first candidate image set and the second candidate image set respectively;

[0021] Based on the matching degrees of images other than the same image in the first candidate image set and the second candidate image set and the target matching degrees of the same images, an image whose matching degree meets a preset matching condition is determined as the target image.

[0022] In some embodiments, for each identical image in the first candidate image set and the second candidate image set, determining a target matching degree of the image based on matching degrees of the image in the first candidate image set and the second candidate image set, respectively, includes:

[0023] For each identical image in the first candidate image set and the second candidate image set, weighting a first matching degree of the image in the first candidate image set using a preset first weight to obtain a first weighted result; and weighting a second matching degree of the image in the second candidate image set using a preset second weight to obtain a second weighted result;

[0024] The sum of the first weighted result and the second weighted result corresponding to the image is determined as the target matching degree of the image.

[0025] In some embodiments, the target image is associated with a degree of match with the description information, and the method further comprises:

[0026] The target images are sorted according to the matching degree and then displayed.

[0027] According to a second aspect of an embodiment of the present disclosure, there is provided an image search device, including:

[0028] An input module configured to obtain description information of an image to be searched;

[0029] An information extraction module, configured to extract key field information from the description information;

[0030] A structured search module, configured to determine a first candidate image set in an image library based on the key field information;

[0031] a semantic search module configured to extract text features in the description information using a preset text-image association model, and determine a second candidate image set in the image library based on the text features;

[0032] The search result fusion module is configured to determine a target image matching the description information based on the first candidate image set and the second candidate image set.

[0033] In some embodiments, each image in the image library is associated with image metadata, and / or each image is associated with an optical character recognition result; the structured search module includes:

[0034] a matching module configured to match the key field information with the image metadata of each image in the image library, and filter out a first matching set from the image library; and / or,

[0035] an index search module configured to match the key field information with an optical character recognition result of each image in the image library, and filter out a second matching set from the image library;

[0036] The first result determination module is configured to determine the first candidate image set according to the first matching set and / or the second matching set.

[0037] In some embodiments, the key field information includes attribute identification information and / or content identification information; wherein the attribute identification information includes at least one of the following: time information, location information.

[0038] In some embodiments, the semantic search module includes:

[0039] A feature calculation module, configured to extract text features in the description information using the text-image association model;

[0040] a feature matching module, configured to match the text feature with the image feature of each image in the image library to obtain a matching result between the description information and each image; wherein the image feature of each image in the image library is obtained based on the text-image association model;

[0041] The second result determination module determines the second candidate image set according to the matching result between the description information and each image.

[0042] In some embodiments, the search result fusion module is further configured to determine, for each identical image in the first alternative image set and the second alternative image set, a target matching degree of the image based on the matching degrees of the image in the first alternative image set and the second alternative image set respectively; based on the matching degrees of each image other than the identical image in the first alternative image set and the second alternative image set, and the target matching degrees of each identical image, determine the image whose matching degree meets the preset matching conditions as the target image.

[0043] In some embodiments, the search result fusion module is further configured to, for each identical image in the first candidate image set and the second candidate image set, weight the first matching degree of the image in the first candidate image set using a preset first weight to obtain a first weighted result; and weight the second matching degree of the image in the second candidate image set using a preset second weight to obtain a second weighted result; and determine the sum of the first weighted result and the second weighted result corresponding to the image as the target matching degree of the image.

[0044] In some embodiments, the target image is associated with a degree of matching with the description information, and the apparatus further comprises:

[0045] The display module is further configured to sort the target images according to the matching degree and then display them.

[0046] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0047] processor;

[0048] a memory for storing processor-executable instructions;

[0049] Wherein, the processor is configured to execute the image search method as described in the first aspect above.

[0050] According to a fourth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0051] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image search method as described in the first aspect above.

[0052] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0053] In an embodiment of the present disclosure, the electronic device does not limit the description information of the image to be searched to be a tag-type phrase, supports the processing of multiple types of description information, and is highly intelligent; in addition, the electronic device combines a first set of candidate images determined based on key field information with a second set of candidate images determined by using text features extracted by a text-image association model to jointly determine a target image that matches the description information, that is, combines the two search methods, which can improve the recall rate and / or accuracy of image search relative to a method of determining the target image based on one search method.

[0054] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0056] Figure 1 It is an architectural diagram of the image search method in the embodiment of the present disclosure.

[0057] Figure 2 It is a flow chart of an image search method shown in an embodiment of the present disclosure.

[0058] Figure 3 It is an example flowchart of the image search method in the embodiment of the present disclosure.

[0059] Figure 4 It is a block diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0061] Figure 1 is an architecture diagram of the image search method in the embodiment of the present disclosure, such as Figure 1 As shown, it includes three parts: the input and output part marked by L1, the core logic processing part marked by L2, and the information extraction part marked by L3. Different parts include different functional modules. For example, the input and output part includes an input module 101 and a search result fusion module 105; the core logic processing part includes an information extraction module 102, a structured search module 103, and a semantic search module 104; the information extraction part is a module for extracting information from images in the image library to be searched. Figure 1 Each module included in the architecture diagram shown can be applied to the image search device of the embodiment of the present disclosure. The functions performed by each module in the image search device are described below:

[0062] An input module 101 is configured to obtain description information of an image to be searched;

[0063] An information extraction module 102 is configured to extract key field information from the description information;

[0064] A structured search module 103, configured to determine a first candidate image set in an image library based on the key field information;

[0065] A semantic search module 104, configured to extract text features in the description information using a preset text-image association model, and determine a second candidate image set in the image library based on the text features;

[0066] The search result fusion module 105 is configured to determine a target image matching the description information based on the first candidate image set and the second candidate image set.

[0067] In some embodiments, each image in the image library is associated with image metadata, and / or each image is associated with an optical character recognition result; the structured search module 103 includes:

[0068] A matching module 103a is configured to match the key field information with the image metadata of each image in the image library, and select a first matching set from the image library; and / or,

[0069] An index search module 103b is configured to match the key field information with an optical character recognition result of each image in the image library, and select a second matching set from the image library;

[0070] The first result determination module is configured to determine the first candidate image set according to the first matching set and / or the second matching set.

[0071] It should be noted that the first candidate image set is Figure 1 The result 1.

[0072] In some embodiments, the semantic search module 104 includes:

[0073] A feature calculation module 104a, configured to extract text features in the description information using the text and image association model;

[0074] A feature matching module 104b is configured to match the text feature with the image feature of each image in the image library to obtain a matching result between the description information and each image; wherein the image feature of each image in the image library is obtained based on the text-image association model;

[0075] The second result determination module determines the second candidate image set according to the matching result between the description information and each image.

[0076] It should be noted that the second candidate image set is Figure 1 The result in 2.

[0077] In some embodiments, the search result fusion module 105 is further configured to determine, for each identical image in the first alternative image set and the second alternative image set, a target matching degree of the image based on the matching degrees of the image in the first alternative image set and the second alternative image set respectively; based on the matching degrees of each image other than the identical image in the first alternative image set and the second alternative image set, and the target matching degrees of each identical image, determine the image whose matching degree meets the preset matching conditions as the target image.

[0078] In some embodiments, the search result fusion module 105 is further configured to, for each identical image in the first candidate image set and the second candidate image set, weight the first matching degree of the image in the first candidate image set using a preset first weight to obtain a first weighted result; and weight the second matching degree of the image in the second candidate image set using a preset second weight to obtain a second weighted result; and determine the sum of the first weighted result and the second weighted result corresponding to the image as the target matching degree of the image. It should be noted that, since the images in the image library are constantly increasing, the information extraction part of the L3 identifier in the disclosed embodiment can use an incremental extraction method to extract features from the newly added images in the image library and save them to the feature library. In addition, considering that feature loading is time-consuming, the disclosed embodiment can also store image features in memory to increase the feature loading speed.

[0079] based on Figure 1The specific manner in which each functional module performs operations in the architecture diagram shown in the figure will be described in detail in the following method embodiments. Figure 2 is a flow chart of an image search method shown in an embodiment of the present disclosure, such as Figure 2 As shown, the image search method includes the following steps:

[0080] S11, obtaining description information of the image to be searched;

[0081] S12, extracting key field information from the description information;

[0082] S13, determining a first candidate image set in the image library based on the key field information;

[0083] S14, extracting text features in the description information using a preset text and image association model, and determining a second candidate image set in the image library based on the text features;

[0084] S15: Determine a target image matching the description information based on the first candidate image set and the second candidate image set.

[0085] The image search method provided by the embodiment of the present disclosure may be implemented independently by a terminal or a server, or may be implemented jointly by a terminal and a server. For example, the terminal may independently perform the image search method described in the embodiment of the present disclosure, or the terminal may send description information of the image to be searched to the server, and the server may execute the image search method according to the received description information of the image to be searched to determine the target image, and return the target image to the terminal.

[0086] In the embodiments of the present disclosure, the terminal may be a smart phone, a tablet computer, a camera, a smart speaker, etc. In addition, the server may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers. The terminal and the server may be connected to each other by wired or wireless communication, which is not limited in the embodiments of the present disclosure. Taking the application of the image search method of the embodiments of the present disclosure to an electronic device as an example, the electronic device includes the aforementioned terminal or server.

[0087] In step S11, the electronic device obtains description information of the image to be searched, and the description information may be text description information and / or image description information. Taking text description information as an example, it may be a description sentence "I had a barbecue with my colleagues in Jingxian last summer", or it may be a description word such as "Christmas", "sports", "scenery", etc. Taking image description information as an example, it may be an image with or without a watermark. It should be noted that the electronic device may obtain the description information of the image to be searched based on the text or image manually input by the user in the search box, and the electronic device may also obtain the description information of the image to be searched based on the user's voice input.

[0088] In step S12, the electronic device extracts key field information from the description information, where the key field information refers to information that can identify the attributes of the image to be searched, and the image search method based on the key field information can also be understood as a structured search method. In some embodiments, the key field information includes attribute identification information and / or content identification information; wherein the attribute identification information includes at least one of the following: time information, location information.

[0089] For example, for text-based description information, key field information may be attribute identification information such as time and place, for example, "last summer" belongs to time, and "Jing County" belongs to place; key field information may also include content identification information, which may be referred to as tags in the disclosed embodiment, for example, key field information may include tags such as "colleagues", "barbecue", "Christmas", "sports", and "scenery". For image-based description information, key field information may be content identification that characterizes the subject of the image, such as tags such as people, animals, or buildings, content identification information that characterizes the scene, such as tags such as sunset, snow scene, or rain scene, or attribute identification information such as time and place included in the image.

[0090] In step S13, the electronic device determines a first candidate image set in the image library based on the key field information. For example, the electronic device can query the image library for candidate images that match the key field information and form a first candidate image set. It should be noted that if the key field information includes multiple information, the first candidate image set can be formed based on the intersection or union of candidate images that match each information. Exemplarily, taking the description information as an example, if the description information includes both text description information and image description information, if the key field information "time", "place", and "label" of the text description information jointly screen out the candidate image subset A, and the key field information "image body" of the image description information screens out the candidate image subset B, then the union of the candidate image subset A and the candidate image subset B can be used as the first candidate image set. Of course, for separate text description information or image description information, the first candidate image set can also be determined by selecting the intersection or union based on the screening results of each key field information.

[0091] The above process of determining the first candidate set based on the key field information corresponds to the function performed by the structured search module 103 .

[0092] In step S14, the electronic device extracts text features in the description information using a preset text-image association model, and determines a second candidate image set in the image library based on the text features. Exemplarily, the text-image association model can be a large model trained based on (Contrastive Language-Image Pre-Training, CLIP), which links visual and language representations to each other to achieve image search tasks. In the disclosed embodiment, the image search method based on text features can be understood as a semantic search method.

[0093] In the disclosed embodiment, the electronic device can use the text-image association model to extract text features (i.e., language features) in the description information, and then associate the text features with the image features (i.e., visual features) of each image in the image library, thereby filtering out the second candidate image set in the image library. It should be noted that the text-image association model can support the processing of text-related information and image-related information at the same time, and the model can automatically call the corresponding processing method based on the input information type.

[0094] Exemplarily, taking the case where the description information includes text-type description information, the text-image association model can extract text features in the text-type description information; taking the case where the description information includes image-type description information, the text-image association model can be used to extract features of the text included in the image-type description information, such as text features of the subject or background of the photo, or text features of a watermark, etc.

[0095] After obtaining the text features in the description information, the electronic device can determine a second set of candidate images in the image library based on the text features. Among them, the image features of each image in the image library can also be extracted using the text and image association model, or can be extracted using methods other than the text and image association model, and the embodiments of the present disclosure do not limit this. It should be noted that the features of each image in the image library can be relatively comprehensive visual features. Taking the use of the text and image association model to extract the image features of each image in the image library and the text features of the image-type description information as an example, for the image-type description information, the features extracted by the text and image association model can focus more on the text in the image, while for the images in the image library, the features extracted by the text and image association model can focus on the visual features expressed by the image content, color, texture, etc. as a whole.

[0096] The above process of extracting text features in the description information based on the image association model and determining the second candidate image set in the image library based on the text features is the function performed by the semantic search module 104 .

[0097] In step S15, the electronic device determines a target image that matches the description information based on the first candidate image set and the second candidate image set, for example, determining the target image as the intersection or union of the images in the first candidate image set and the second candidate image set. In addition, the target image can also be determined based on the matching degree between each image in the first candidate image set and the second candidate image set and the description information. The target image can be one or more images.

[0098] The above process of determining the target image matching the description information based on the first candidate image set and the second candidate image set corresponds to the function performed by the search result fusion module 105 .

[0099] In the related art, taking user terminals as an example, current mainstream equipment manufacturers have all used a method of labeling images to facilitate user image searches. For example, when a user has a lot of image data, if he needs to find a photo, the user can enter tags such as "sports" and "landscape" in the search box, and then the user terminal outputs matching results based on tag search. However, this method has the following two problems: (1) The supported tags are limited, and the matching algorithm cannot exhaust all tags. In this case, the search will result in no tag matches, and the user needs to fall back to "manually flipping through photos" to search; (2) The supported description methods are limited. In addition to using tags, users are usually accustomed to using natural language to describe the content of the picture. The description is not limited to a noun, but may also be a verb, scene, or pronoun.

[0100] In contrast, in the disclosed embodiment of the present invention, the electronic device does not limit the description information of the image to be searched to be a tag-type phrase, supports the processing of multiple types of description information, and is highly intelligent; in addition, the electronic device combines a first set of candidate images determined based on key field information with a second set of candidate images determined by text features extracted using a text-image association model to jointly determine a target image that matches the description information, that is, combines the two search methods, which can improve the recall rate and accuracy of image search relative to a method of determining the target image based on one search method.

[0101] In some embodiments, each image in the image library is associated with image metadata, and / or each image is associated with an optical character recognition result;

[0102] The determining a first candidate image set in an image library based on the key field information includes:

[0103] Matching the key field information with the image metadata of each image in the image library, and selecting a first matching set from the image library; and / or,

[0104] Matching the key field information with an optical character recognition result of each image in the image library, and selecting a second matching set from the image library;

[0105] The first candidate image set is determined according to the first matching set and / or the second matching set.

[0106] In the disclosed embodiment, the image metadata associated with each image in the image library is metadata fixed when the image is generated, such as the time and location of the shooting, etc., and may also include tags obtained after the image is recognized. The optical character recognition (OCR) result associated with each image in the image library may be the optical character recognition result corresponding to the subject text or background text in the image, or the optical character recognition result of the watermark text. It should be noted that the image metadata and / or optical character recognition result associated with each image in the image library can be extracted in advance and stored in the electronic device.

[0107] In the disclosed embodiment, the electronic device may match the key field information with the image metadata of each image in the image library, thereby filtering out a first matching set from the image library, and may also match the key field information with the optical character recognition result of each image, thereby filtering out a second matching set from the image library. It should be noted that when filtering the first matching set or the second matching set, taking the first matching set as an example, based on the matching degree between the key field information and the image metadata corresponding to each image, the image whose matching degree meets the matching degree threshold can be determined as an image in the first matching set. Similarly, the second matching set can also be filtered in a similar manner.

[0108] After determining the first matching set and / or the second matching set, the electronic device can determine the first candidate image set based on the first matching set and / or the second matching set. For example, the union of the first matching set and the second matching set can be determined as the first candidate image set. Of course, the intersection of the first matching set and the second matching set can also be determined as the first candidate image set.

[0109] It can be understood that in the embodiment of the present disclosure, based on the key field information extracted from the description information, the information associated with each image in the image library is used for matching to determine the first candidate image set, and the solution is simple and effective.

[0110] In some embodiments, the extracting text features in the description information using a preset text-image association model, and determining a second candidate image set in the image library based on the text features, includes:

[0111] Extracting text features in the description information by using the text-image association model, and matching the text features with image features of each image in the image library to obtain a matching result between the description information and each image; wherein the image features of each image in the image library are obtained based on the text-image association model;

[0112] The second candidate image set is determined according to the matching result between the description information and each image.

[0113] As mentioned above, the image features of each image in the image library can also be obtained based on the text-image association model, but the image features in the image library are more comprehensive visual features. In this regard, in the embodiment of the present disclosure, the text features in the description information are extracted by using the text-image association model, and the second candidate image set is determined by matching the text features of each image in the image library extracted by the text-image association model. Since the text-image association model (such as the CLIP model) is usually obtained by training on large-scale unlabeled data based on contrastive learning and pre-training, the generalization ability is good. Therefore, the embodiment of the present disclosure compares the text features and the image features based on the same text-image association model to screen the images, and a more accurate result can be obtained. It should be noted that in the embodiment of the present disclosure, the matching result of the description information and each image can be the cosine similarity between the text features and the image features. When the electronic device determines the second candidate image set according to the matching result of the description information and each image, it can select a number of images with the highest cosine similarity (TOP N) as the images in the second candidate image set, or it can select images with a cosine similarity greater than the similarity threshold as the images in the second candidate set.

[0114] In some embodiments, each image in the first candidate image set and the second candidate image set is associated with a matching degree, and determining a target image matching the description information based on the first candidate image set and the second candidate image set includes:

[0115] For each identical image in the first candidate image set and the second candidate image set, determining a target matching degree of the image based on matching degrees of the image in the first candidate image set and the second candidate image set respectively;

[0116] Based on the matching degrees of images other than the same image in the first candidate image set and the second candidate image set and the target matching degrees of the same images, an image whose matching degree meets a preset matching condition is determined as the target image.

[0117] In the disclosed embodiment, each image in the first candidate image set and the second candidate image set is associated with a matching degree, which can be characterized as a matching degree with the description information. The greater the matching degree, the more likely the image is to belong to the target image.

[0118] In the disclosed embodiment, since the same image in the image library may appear in different candidate image sets, and since different candidate image sets are obtained in different ways, the matching degrees of the images in different candidate image sets will also be different accordingly. For example, the matching degree of image M in the first candidate image set is 0.7, and the matching degree of image M in the second candidate image set is 0.5. Then, in order to facilitate the screening of target images based on the matching degree, a unique matching degree must be determined for one image. In this regard, the disclosed embodiment first determines the target matching degree of the image based on the matching degrees of the image in the first candidate image set and the second candidate image set, respectively, for the same image that appears in different candidate image sets. For example, the maximum value or minimum value in the first candidate image set and the second candidate image set may be determined as the target matching degree, or the target matching degree may be calculated based on a predetermined calculation formula.

[0119] In the embodiment of the present disclosure, after determining the target matching degree of the same image appearing in different candidate image sets, the electronic device can combine the matching degrees corresponding to each image other than the same image, and determine the image whose matching degree meets the preset matching condition as the target image, wherein the preset matching condition can be the TOP N images with the highest matching degree, or can be that the matching degree is greater than the matching degree threshold. It can be understood that in the embodiment of the present disclosure, for the same image appearing in different candidate image sets, the target matching degree of the image is re-determined, and then the target image is determined in combination with the matching degrees of each image other than the same image, which can reduce the impact of the matching degree of the same image in different candidate image sets on the determination of the target image, and help improve the accuracy of the determination of the target image.

[0120] In some embodiments, for each identical image in the first candidate image set and the second candidate image set, determining a target matching degree of the image based on matching degrees of the image in the first candidate image set and the second candidate image set, respectively, includes:

[0121] For each identical image in the first candidate image set and the second candidate image set, weighting a first matching degree of the image in the first candidate image set using a preset first weight to obtain a first weighted result; and weighting a second matching degree of the image in the second candidate image set using a preset second weight to obtain a second weighted result;

[0122] The sum of the first weighted result and the second weighted result corresponding to the image is determined as the target matching degree of the image.

[0123] As mentioned above, the target matching degree of the same image appearing in different candidate image sets can be calculated based on a predetermined calculation formula. In the embodiment of the present disclosure, the electronic device weights and sums the corresponding matching degrees based on the preset first weight corresponding to the first candidate image set and the preset second weight corresponding to the second candidate image set, thereby obtaining the target matching degree. Exemplarily, if the preset first weight corresponding to the first candidate image set is 0.7 and the preset second weight corresponding to the second candidate image set is 0.9, then the target weight of the aforementioned image M is 0.94 (0.7*0.7+0.9*0.5).

[0124] It should be noted that, in the disclosed embodiment, the preset first weight and the preset second weight actually correspond to the weights corresponding to the acquisition methods of different candidate image sets. Among them, the preset first weight can be understood as the weight corresponding to the structured search method; and the preset second weight corresponds to the weight corresponding to the semantic search method based on the large model. The preset first weight and the preset second weight can be weights determined by the tester after testing and adjusting based on a large amount of sample data.

[0125] It can be understood that in the embodiment of the present disclosure, the electronic device weights the matching degree of the same image in different candidate image sets based on the preset first weight and the preset second weight to obtain the target matching degree, which enables the image to integrate the matching performance of different image search methods, and helps to improve the accuracy of the final determination of the target image.

[0126] In some embodiments, the target image is associated with a degree of match with the description information, and the method further comprises:

[0127] The target images are sorted according to the matching degree and then displayed.

[0128] In the disclosed embodiment, as mentioned above, the target image may include multiple images, and the target images are sorted and displayed according to the matching degree, for example, displayed in order from high to low matching degree, which can facilitate users to see more matching images first, has good intelligence, and can enhance user experience.

[0129] Based on the above description, exemplary, Figure 1In the example, the electronic device can obtain the text description information of "having a barbecue with colleagues in Jing County last summer" based on the user's input in the search box, extract information from the obtained description information, and obtain regular information such as "season / year / month / day / festival / province / city" (i.e., key field information). Then, the structured search module 103 matches the time / place in the extracted regular information with the meta information (time and place, etc.) of each picture in the image library, and matches the person or object information in the extracted regular information with the tag information (person / object) of each picture in the image library. The first matching set is obtained based on the matching of each field information, and the exact matching method or the fuzzy matching method can be used in the matching process. In addition, the structured search module 103 also performs an index search based on the extracted regular information and the text information recognized by OCR for each picture in the image library to obtain a second matching set. The electronic device can determine the result 1 output by the structured search module 103 based on the first matching set and the second matching set. In addition, the semantic search module 104 extracts text features of the description information by loading the text and image association model, and then performs feature matching of the text features of the description information with the image features of each picture in the image library to obtain result 2. The electronic device can fuse result 1 and result 2 through the search result fusion module 105, and select the TOP N images as the target images.

[0130] Figure 3 is a flowchart of an image search method in an embodiment of the present disclosure. Figure 3 As shown, the following steps are included:

[0131] 1. The user enters a search description.

[0132] Based on the search description input by the user, the electronic device can obtain description information of the image to be searched.

[0133] 2. The electronic device extracts structural information based on the information extraction module: time, place, etc.

[0134] 3.1. Electronic device retrieval metadata.

[0135] 3.2. Electronic device retrieval of text.

[0136] 4. The electronic device obtains search result 1.

[0137] In this embodiment, the structural information is the key field information. The electronic device performs metadata retrieval based on the extracted key field information, that is, matches the key field information with the image metadata of each image in the image library, and filters out a first matching set from the image library; in addition, the electronic device performs text information retrieval based on the extracted key field information, that is, matches the key field information with the optical character recognition results of each image in the image library, and filters out a second matching set from the image library. In step 4, the electronic device obtains retrieval result 1, that is, determines a first candidate image set based on the first matching set and the second matching set. The above process is Figure 1 The process performed by the structured search module 103.

[0138] 5. Electronic equipment dynamically loads large models.

[0139] 6. Electronic devices calculate text features through large models.

[0140] 7. The electronic device performs feature matching through the large model to obtain result 2.

[0141] In this embodiment, the large model is the text and image association model of the embodiment of the present disclosure, and the process of steps 6 and 7 is the process of the electronic device using the preset text and image association model to extract text features in the description information and determine the second candidate image set in the image library based on the text features, that is, Figure 1 The process performed by the semantic search module 104.

[0142] 8. The electronic device obtains search result 1 and result 2, and performs data fusion.

[0143] In the embodiment of the present disclosure, the electronic device obtains search result 1 and result 2 and performs data fusion, that is, a process of determining a target image matching the description information based on the first candidate image set and the second candidate image set, that is, Figure 1 The process performed by the search result fusion module 105.

[0144] 9. Results display.

[0145] In an implementation of the present disclosure, the electronic device may display the target images in order based on the matching degrees of the target images.

[0146] It should be noted that in the embodiment of the present disclosure, the information extraction module in the electronic device will automatically and regularly trigger the extraction of information on metadata / text data of images in the image library, as well as the extraction of image feature data.

[0147] It can be understood that in the embodiments of the present disclosure, the electronic device does not limit the description information of the image to be searched to be a tag-type phrase, but supports any form of retrieval description, which is highly intelligent. In addition, the electronic device combines the two search methods, which can improve the recall rate and / or accuracy of the image search compared to the method of determining the target image based on one search method.

[0148] Figure 4 8 is a block diagram of an electronic device according to an exemplary embodiment. The electronic device may be identified as a device 800, which may be a smart phone, a tablet computer, etc.

[0149] Reference Figure 4 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0150] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0151] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0152] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0153] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0154] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0155] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0156] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0157] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0158] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0159] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0160] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the aforementioned image search method.

[0161] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims above.

[0162] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image search method, characterized in that: The method comprises: Get the description information of the image to be searched; Extract key field information from the description information; Determining a first candidate image set in an image library based on the key field information; Extracting text features in the description information using a preset text-image association model, and determining a second candidate image set in the image library based on the text features; Based on the first candidate image set and the second candidate image set, a target image matching the description information is determined.

2. The method according to claim 1, characterized in that Each image in the image library is associated with image metadata, and / or each image is associated with an optical character recognition result; The determining a first candidate image set in an image library based on the key field information includes: Matching the key field information with the image metadata of each image in the image library, and selecting a first matching set from the image library; and / or, Matching the key field information with an optical character recognition result of each image in the image library, and selecting a second matching set from the image library; The first candidate image set is determined according to the first matching set and / or the second matching set.

3. The method according to claim 1 or 2, characterized in that: The key field information includes attribute identification information and / or content identification information; wherein the attribute identification information includes at least one of the following: time information, location information.

4. The method according to claim 1, characterized in that: The extracting text features in the description information by using a preset text-image association model, and determining a second candidate image set in the image library based on the text features, includes: Extracting text features in the description information by using the text-image association model, and matching the text features with image features of each image in the image library to obtain a matching result between the description information and each image; wherein the image features of each image in the image library are obtained based on the text-image association model; The second candidate image set is determined according to the matching result between the description information and each image.

5. The method according to claim 1 or 4, characterized in that: Each image in the first candidate image set and the second candidate image set is associated with a matching degree, and determining a target image matching the description information based on the first candidate image set and the second candidate image set includes: For each identical image in the first candidate image set and the second candidate image set, determining a target matching degree of the image based on matching degrees of the image in the first candidate image set and the second candidate image set respectively; Based on the matching degrees of images other than the same image in the first candidate image set and the second candidate image set and the target matching degrees of the same images, an image whose matching degree meets a preset matching condition is determined as the target image.

6. The method according to claim 5, characterized in that The determining, for each identical image in the first candidate image set and the second candidate image set, a target matching degree of the image based on matching degrees of the image in the first candidate image set and the second candidate image set, respectively, comprises: For each identical image in the first candidate image set and the second candidate image set, weighting a first matching degree of the image in the first candidate image set using a preset first weight to obtain a first weighted result; and weighting a second matching degree of the image in the second candidate image set using a preset second weight to obtain a second weighted result; The sum of the first weighted result and the second weighted result corresponding to the image is determined as the target matching degree of the image.

7. The method according to claim 1, characterized in that The target image is associated with a degree of matching with the description information, and the method further comprises: The target images are sorted according to the matching degree and then displayed.

8. An image search device, characterized in that: The device comprises: An input module configured to obtain description information of an image to be searched; An information extraction module, configured to extract key field information from the description information; A structured search module, configured to determine a first candidate image set in an image library based on the key field information; a semantic search module configured to extract text features in the description information using a preset text-image association model, and determine a second candidate image set in the image library based on the text features; The search result fusion module is configured to determine a target image matching the description information based on the first candidate image set and the second candidate image set.

9. The device according to claim 8, characterized in that Each image in the image library is associated with image metadata, and / or each image is associated with an optical character recognition result; the structured search module includes: a matching module configured to match the key field information with the image metadata of each image in the image library, and filter out a first matching set from the image library; and / or, an index search module configured to match the key field information with an optical character recognition result of each image in the image library, and filter out a second matching set from the image library; The first result determination module is configured to determine the first candidate image set according to the first matching set and / or the second matching set.

10. The device according to claim 8 or 9, characterized in that The key field information includes attribute identification information and / or content identification information; wherein the attribute identification information includes at least one of the following: time information, location information.

11. The device according to claim 8, characterized in that The semantic search module includes: A feature calculation module, configured to extract text features in the description information using the text-image association model; a feature matching module, configured to match the text feature with the image feature of each image in the image library to obtain a matching result between the description information and each image; wherein the image feature of each image in the image library is obtained based on the text-image association model; The second result determination module determines the second candidate image set according to the matching result between the description information and each image.

12. The device according to claim 8 or 11, characterized in that The search result fusion module is further configured to determine, for each identical image in the first candidate image set and the second candidate image set, a target matching degree of the image based on matching degrees of the image in the first candidate image set and the second candidate image set respectively; Based on the matching degrees of images other than the same image in the first candidate image set and the second candidate image set and the target matching degrees of the same images, an image whose matching degree meets a preset matching condition is determined as the target image.

13. The device according to claim 12, characterized in that The search result fusion module is further configured to weight, for each identical image in the first candidate image set and the second candidate image set, a first matching degree of the image in the first candidate image set using a preset first weight to obtain a first weighted result; And use a preset second weight to weight the second matching degree of the image in the second candidate image set to obtain a second weighted result; and determine the sum of the first weighted result and the second weighted result corresponding to the image as the target matching degree of the image.

14. The device according to claim 8, characterized in that The target image is associated with a degree of matching with the description information, and the device further comprises: The display module is configured to sort the target images according to the matching degree and then display them.

15. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the image search method as described in any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image search method as described in any one of claims 1 to 7.

Images