Picture search method, electronic device, storage medium, and program product

Abstract

Embodiments of the present application provide a picture search method, an electronic device, a storage medium and a program product, comprising: in response to receiving a picture search request, obtaining a text semantic feature and a keyword of a search text, the search text being carried in the picture search request; in the case of obtaining the keyword, searching a picture information set respectively matched with the text semantic feature and at least one keyword from a picture information library to obtain a candidate picture information set; and selecting a candidate picture information satisfying a preset condition from the candidate picture information set as a picture search result according to a similarity corresponding to each candidate picture information in the candidate picture information set, the similarity being a similarity between a picture global feature of the corresponding candidate picture information and the text semantic feature. The scheme can improve the picture search efficiency, reduce the calculation overhead, and also take into account the recall rate and accuracy of the picture search, and reduce the index persistence cost of the semantic vector.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image search method, electronic device, storage medium, and program product. Background Technology

[0002] With the advent of the digital age, cloud storage products such as cloud drives have become important tools for people to store and manage images, text, and other data. As the number of users and usage scenarios increase, the diversity and sheer scale of content in cloud storage pose new challenges to image retrieval.

[0003] Currently, cloud storage products mainly rely on text for image search. However, existing technologies require a large amount of computation for image search, which greatly reduces retrieval efficiency and increases computing costs. Summary of the Invention

[0004] This application provides an image search method, an electronic device, a computer-readable storage medium, and a computer program product to alleviate or solve one or more technical problems existing in the prior art.

[0005] In a first aspect, embodiments of this application provide an image search method, the method comprising: in response to receiving an image search request, acquiring textual semantic features and keywords of search text, wherein the search text is carried in the image search request and the keywords are words related to the search intent; if the keywords are acquired, based on the image global features of each image stored in an image information database and textual description information related to the image, searching from the image information database for a set of image information that matches the textual semantic features and the at least one keyword respectively, to obtain a set of candidate image information, wherein the image information database stores the correspondence between image identification information, image global features and image description information; and according to the similarity of each candidate image information in the candidate image information set, selecting candidate image information that meets preset conditions as the image search result, wherein any similarity refers to the similarity between the image global features of the corresponding candidate image information and the textual semantic features.

[0006] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0008] Fourthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0009] Using the method provided in this application embodiment, when performing image search, after obtaining the text semantic features and keywords of the search text, based on the global image features of each image stored in the image information database and the text descriptions related to the images, a set of image information matching the text semantic features and the keywords is searched from the image information database. Then, according to the similarity of each candidate image information in the candidate image information set, the candidate image information is filtered. By combining preliminary recall and secondary filtering, the accuracy of image search is improved. Furthermore, in this solution, image search and similarity calculation are performed in the image information database. In this application, global image feature vectors are used throughout the search process. Compared with related technologies that use both global and local image features, this significantly reduces the amount of feature data. On the one hand, it reduces the amount of data stored in the image database, saving storage space and lowering the cost of persisting the semantic vector index. On the other hand, it also reduces the computational load during image search, improving both search efficiency and computational cost. Furthermore, in this application, image searches are performed based on two different dimensions: the semantic features of the search text in the search request and the keywords of the search text. This ensures the comprehensiveness of the image search and improves the image recall rate.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0012] Figure 1 This illustration shows an application scenario diagram of the image search method provided in the embodiments of this application;

[0013] Figure 2 This illustration shows one of the flowcharts of the image search method provided in an embodiment of this application;

[0014] Figure 3 An exemplary computational flow graph of the BERT model in the image search method provided in this application is shown;

[0015] Figure 4 The second schematic flowchart of the image search method provided in the embodiment of this application is shown;

[0016] Figure 5 The third schematic flowchart of the image search method provided in the embodiment of this application is shown;

[0017] Figure 6 This paper shows a schematic diagram of the module composition of the image search device provided in an embodiment of this application;

[0018] Figure 7 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0020] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0021] With the advent of the digital age, cloud storage products such as cloud drives have become important tools for people to store and manage images, text, and other data. As the number of users and usage scenarios increase, the diversity and sheer scale of content in cloud storage pose new challenges to image retrieval.

[0022] Currently, image retrieval in cloud storage can be performed using text descriptions. This is achieved by matching the text feature vector of the user-inputted text with the feature vectors of images in the cloud storage. A related technology matches the image's text feature vector with both global and local feature vectors. However, local feature vectors contain a large amount of data, requiring significant computation for feature vector matching, thus increasing computational costs and reducing image retrieval efficiency. This is especially true in scenarios with large-scale image storage, where the amount of data to be matched surges, further increasing the computational burden, reducing retrieval efficiency, and placing higher demands on computing device performance.

[0023] In view of this, this application provides an image search method. When performing an image search, after obtaining the textual semantic features and keywords of the search text, based on the global image features of each image stored in the image database and the textual description information related to the image, a set of image information matching the textual semantic features and at least one keyword is searched from the image database to obtain a candidate image information set. Then, the candidate image information is filtered according to the similarity of each candidate image information in the candidate image information set. By combining preliminary recall and secondary filtering, the accuracy of image search is improved. Furthermore, it uses a smaller image global feature vector, which significantly reduces the amount of feature data compared to related technologies that use both global and local image features. This reduces the amount of data stored in the image database, saving storage space, and also reduces the computational load during image search, improving both efficiency and computational cost. In addition, in this application, image search is performed based on two different dimensions: the textual semantic features of the search text and the keywords of the search text, ensuring the comprehensiveness of the image search and improving the image recall rate.

[0024] To facilitate understanding of the embodiments of this application, the application scenarios of the image search method provided in the embodiments of this application will be briefly described first. Figure 1 The illustration shows an application scenario diagram of the image search method provided in the embodiments of this application. For example... Figure 1 As shown, this application scenario includes a server 110 and a client 120, with the client 120 communicating with the server 110. The client 120 can be deployed on computing devices such as mobile phones, computers, and tablets. The server 110 can be a cloud storage server, and the client 120 can be a cloud storage client.

[0025] In one application example, when a user needs to search for images in a cloud drive, they can enter search text on client 120. Client 120 generates an image search request based on the user's input search text and sends it to server 110. Server 110 has an image database deployed on it, storing the correspondence between image identifiers, global image features, image descriptions, and image metadata. The image identifiers can be unique codes or symbols that identify an image. After receiving the image search request from client 120, server 110 obtains the semantic features and keywords of the search text, which are carried in the image search request. The keywords are words related to the search intent. Alternatively, in another implementation, after receiving the user's input search text, client 120 generates the corresponding semantic features and keywords and sends them to server 110 for image search. Once the keywords are obtained, based on the global image features of each image stored in the image information database and the text description information related to the images, the image information database is searched for image information sets that match the aforementioned text semantic features and at least one keyword, respectively. The image information sets corresponding to the text semantic features are merged with the image information sets corresponding to at least one keyword to obtain a candidate image information set.

[0026] The aforementioned candidate image information set is a collection of candidate image information initially recalled from the text semantic feature dimension and the keyword dimension, respectively. To further improve accuracy, image information that meets preset conditions is selected from the candidate image information set according to the similarity corresponding to each candidate image information. Here, any similarity refers to the similarity between the global features of the corresponding candidate image information and the aforementioned text semantic features. The preset conditions can be image information with a similarity value greater than or equal to a certain threshold, or a certain number of image information truncated according to the similarity value from largest to smallest, or a combination of both. When implementing this scheme, the preset conditions can be set according to the actual application scenario. This is merely an example listing several possible implementation forms of the preset conditions and is not a limitation of this application. The corresponding image is queried based on the image identifier information in the selected image information, and the image retrieval result is returned to the client 120.

[0027] It should be noted that the application scenarios or examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0028] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0029] Figure 2 This document illustrates one of the flowcharts of the image search method provided in an embodiment of this application, such as... Figure 2 As shown, this method can be applied to Figure 1 Server 110 in the illustrated application scenario. Among them, Figure 2 The method shown may include steps S201, S202 and S203.

[0030] Step S201: In response to receiving an image search request, obtain the text semantic features and keywords of the search text, which are carried in the image search request. The keywords are words related to the search intent.

[0031] Step S202: If the keywords are obtained, based on the global features of each image stored in the image information database and the text description information related to the images, search the image information database for image information sets that match the text semantic features and at least one keyword respectively, and obtain the candidate image information set.

[0032] Step S203: Based on the similarity of each candidate image in the candidate image information set, select candidate image information that meets the preset conditions as the image search results. Any similarity refers to the similarity between the global features of the corresponding candidate image information and the semantic features of the text.

[0033] In this implementation, the image search request is sent by the client. When a user needs to search for an image, they can enter the corresponding search text in an input box on the client interface. This search text describes information related to the image the user wants to search for; for example, the user's search text could be "a yellow cat photographed in Hangzhou on May 11, 2021". After entering the search text, the user clicks the corresponding control on the interface, such as a "submit" or "send" control. The client responds to the user's click by generating an image search request based on the received search text and sending it to the server. For example, if the user is searching for images on a cloud storage service, the client is a cloud storage client, and the server is a cloud storage server.

[0034] When the server receives an image search request from a client, it extracts the search text carried in the request and obtains the textual semantic features and keywords of the search text. The aforementioned textual semantic features refer to the textual characteristic vector of the search text, which can be extracted using a feature extraction model. In one implementation, the search text can be feature extracted using the first tower model in a dual-tower model used for textual feature vector extraction.

[0035] The keywords mentioned above refer to words related to the search intent. These can be information related to the image, such as the location and time the image was taken, or information related to the objects contained in the image, such as the name and characteristics of the objects. For example, if the user enters the search text "a yellow cat photographed in Hangzhou on May 11, 2021", then the keywords corresponding to this search text could be "May 11, 2021", "Hangzhou", "yellow", and "cat".

[0036] It should be noted that when implementing this solution, words related to the search intent in the search text can be directly used as the keywords mentioned above, or the words related to the search intent in the search text can be pre-processed before being used as the keywords. For example, this pre-processing can include format conversion, vocabulary expansion, or semantic conversion. In practical applications, appropriate preprocessing methods can be selected according to actual needs. The examples provided here are merely for ease of understanding and are not intended to limit the scope of this solution.

[0037] In one implementation, a global feature vector of the image can be extracted using a feature extraction model. For example, the second tower model in a dual-tower model, used for image feature vector extraction, can be used to extract features from the image. That is, in this embodiment, the two towers of the dual-tower model extract features from the image and the search text respectively, unifying the extracted text semantic features and image global features within the same semantic space. This allows the obtained text semantic features and image global features to be directly matched subsequently without additional semantic space mapping processing, thereby improving processing efficiency.

[0038] If keywords are obtained, a candidate image information set is obtained through step S202. For example, in step S202, the text semantic features are matched with the global features of each image in the image database to obtain a preliminary image information set that matches the text semantic features. Then, images matching the shooting time and location requirements in the keywords are selected from this preliminary image information set. Additionally, the keywords are matched with the image descriptions in the image database to obtain an image information set that matches at least one text semantic feature. The two image information sets obtained are then merged to obtain the candidate image information set. This candidate image information set considers not only the text description dimension but also the semantic feature vector dimension, ensuring the comprehensiveness and accuracy of image retrieval. The candidate image information set obtained in this way can more comprehensively cover image information related to the image search request, thereby improving the relevance and coverage of the search results.

[0039] Step S202 above considers the comprehensiveness of image retrieval. However, considering that step S202 may include some image information that is not highly relevant to the user's search intent, the candidate image information set obtained in step S202 can be further filtered to ensure that the images presented to the user are highly relevant to the search intent, that is, to ensure the accuracy of image search.

[0040] In step S203 above, the similarity between the global image features and the text semantic features of each candidate image in the candidate image set can be calculated, and a secondary filtering of each candidate image in the candidate image set can be performed based on the similarity. For example, candidate image information with a similarity greater than or equal to a preset threshold can be filtered as search results, or the candidate image information in the candidate image set can be sorted according to similarity, and a preset number of images can be extracted as search results in descending order of similarity.

[0041] In one implementation, the similarity between the global image features and the textual semantic features of each candidate image in the candidate image set can be calculated using a neural network model. For example, a neural network model constructed using a transformer network and fully connected layers can be used to calculate this similarity.

[0042] In this embodiment, when performing image search, after obtaining the keywords of the search text, based on the global image features of each image stored in the image information database and the text description information related to the image, a set of image information matching the semantic features of the text and the at least one keyword is searched from the image information database. Then, the candidate image information is filtered according to the similarity of each candidate image information in the candidate image information set. By combining preliminary recall and secondary filtering, the accuracy of image search is improved. In this solution, both image global feature vectors are used when performing image search and calculating similarity in the image information database. Compared with the use of image global features and image local features in related technologies, the amount of feature data is greatly reduced. On the one hand, it can reduce the amount of data stored in the image information database, save storage space, and reduce the index persistence cost of semantic vectors. On the other hand, it can also reduce the amount of computation during image search, which can improve image search efficiency and reduce computational overhead. In addition, in this solution, when performing image search, image search is performed based on two different dimensions: the semantic features of the search text in the image search request and the keywords of the search text. This ensures the comprehensiveness of image search and improves image recall rate.

[0043] Additionally, it should be noted that in this embodiment of the application, if no keywords are obtained from the search text, only a set of image information that matches the semantic features of the text is searched from the image information database as the candidate image information set.

[0044] In one implementation, the keywords include search intent descriptors in the search text and target tags matching the search text. The search intent descriptors include at least one of shooting time information, shooting location information, and object information contained in the image. The target tags are used to characterize object description information of the objects contained in the image. In this case, in step S202, based on the global image features of each image stored in the image information database and the text description information related to the image, searching the image information database for image information sets that match the text semantic features and keywords respectively, to obtain a candidate image information set, may include the following steps: searching the image information database for a first image information set that matches the search intent descriptors and target tags, and searching the image information database for a second image information set that matches the text semantic features and search intent descriptors; merging the first image information set and the second image information set to obtain the candidate image information set.

[0045] In one implementation, the search intent descriptor and target tag can be matched with each text description information stored in the image information database to obtain a set of matching image information, which serves as the first image information set. Furthermore, the search intent descriptor is matched with each text description information stored in the image information database, and the text semantic features are matched with the global image features stored in the image information database. The set of image information that matches both the search intent descriptor and the text semantic features serves as the second image information set.

[0046] That is, in this embodiment of the application, each image information in the first image information set needs to match both the search intent descriptor and the target tag. Each image information in the second image information set needs to match both the text semantic features and the search intent descriptor.

[0047] In this embodiment, global image features represent the image from the dimension of feature vectors, while image description information and image metadata represent the image from the dimension of key information description. To improve the recall rate of image retrieval, this solution performs retrieval in the image information database from the dimensions of textual key information and feature vectors respectively, and merges the image information sets retrieved from the two dimensions to obtain a candidate image information set.

[0048] In this embodiment, when searching for image information in an image database based on the semantic features and keywords corresponding to the search text, matching the semantic features can identify image information that has a high semantic match with the search intent. Matching the target tags can quickly locate image information that matches the target tags. Both dimensions of retrieval must simultaneously meet the requirements of the search intent descriptive words in the search text. The retrieval results of these two dimensions complement each other. Thus, even if one dimension fails to fully cover the user's search intent, the other dimension can provide supplementary information, thereby significantly improving the recall rate of image retrieval and ensuring that users can obtain more comprehensive image search results.

[0049] In this embodiment, the image-related text description information stored in the image information database includes image description information and image metadata. For example, the image identification information, image global features, image description information, and image metadata can be stored in the image information database in a one-to-one correspondence. The image description information represents text description information related to the image, such as information about objects contained in the image and feature information of those objects. The image metadata can be information about the image's shooting time and location. The image global features are used to characterize the overall appearance and content of the image, such as color features, texture features, or shape features.

[0050] Therefore, in this embodiment of the application, the above-mentioned search for a first set of image information matching the search intent descriptor and the target tag from the image information database may include the following steps: matching the search intent descriptor with each image metadata in the image information database to obtain a first list of image information matching the search intent descriptor; matching the target tag with each image description in the image information database to obtain a second list of image information matching the target tag; and obtaining a first set of image information based on the image information that coexists in the first and second image information lists.

[0051] In one implementation, for any image description in the image database, it needs to be matched against the aforementioned target tags. If the image description contains the aforementioned target tags, then the image identifier information corresponding to that image description is added to the corresponding image information set. For example, the number of target tags can be one or more. Therefore, "the image description contains target tags" can refer to the image description containing all target tags of the search text.

[0052] Furthermore, all image metadata in the image database needs to be matched against the aforementioned search intent descriptors. These image metadata may contain multiple metadata entries, such as image capture time and location. Therefore, when matching at least one keyword with the image description information, each search intent descriptor can be matched against its corresponding metadata to determine if the metadata matches the search intent descriptor's requirements. For example, regarding capture time, it can be determined whether the capture time in the image metadata falls within the time range described by the search intent descriptor; if so, the search intent descriptor is considered a match. Similarly, regarding capture location, it can be determined whether the capture location in the image metadata falls within the location range described by the search intent descriptor; if so, the search intent descriptor is considered a match.

[0053] Additionally, it should be noted that in this embodiment of the application, if there are multiple search intent descriptors, the above-mentioned image metadata matching the search intent means that the image metadata matches each of the search intent descriptors.

[0054] In this embodiment, the matching of search intent descriptors and target tags can be performed simultaneously to obtain two lists of image information, and the intersection of the two lists can be used as the first image information set. Alternatively, the matching of search intent descriptors and target tags can be performed sequentially. For example, the matching of search intent descriptors can be performed first to obtain the first list of image information, and then the target tags can be matched with the image descriptions in the first list to obtain the first image information set. Alternatively, the matching of target tags can be performed first to obtain the second list of image information, and then the search intent descriptors can be matched with the image clusters in the second list to obtain the first image information set. When implementing this scheme, the corresponding matching logic can be set according to the actual situation. This is only an example of several possible implementation methods and is not intended to limit the embodiments of this application.

[0055] In this embodiment of the application, when searching for the first set of image information from the image information database, the corresponding image information lists are matched from the image information database based on the search intent descriptor and the target tag, respectively. The image information that appears in both image information lists at the same time is taken as the first set of image information. By matching the search intent descriptor and the target tag at the same time, images related to the user's needs can be located more accurately, reducing the interference of irrelevant results, thereby improving the accuracy of image search.

[0056] In one embodiment, the image-related text description information stored in the image information database includes image metadata. The process of searching the image information database for a second set of image information that matches text semantic features and search intent descriptors may include the following steps: matching the text semantic features with the global features of each image in the image information database to obtain a third list of image information that matches the text semantic features; matching the search intent descriptors with the metadata of each image in the image information database to obtain a first list of image information that matches the search intent descriptors; and obtaining a second set of image information based on the image information shared by the first and third image information lists.

[0057] In one implementation, the above-mentioned matching of text semantic features with each global feature of an image in the image information database can be achieved by calculating the vector similarity between the text semantic features and each global feature of an image, and finding the matching image information based on the similarity between the two vectors.

[0058] The specific implementation process of matching the search intent descriptor with the metadata of each image in the image information database to obtain the first list of image information that matches the search intent descriptor can be referred to the above description, and will not be repeated here.

[0059] In this embodiment, when searching for a second set of image information from the image information database, corresponding image information lists are matched from the image information database based on search intent descriptors and text semantic features, respectively. The image information that appears in both image information lists is used as the second set of image information. By combining the dual matching method of search intent descriptors and text semantic features, we can effectively compensate for the result interference problem that may be caused by relying solely on text semantic features for searching, significantly improve the accuracy and relevance of image search, and thus enhance the accuracy of image search.

[0060] In this embodiment of the application, in order to further improve the accuracy of image retrieval, the above-mentioned text semantic features include the global semantic features of the text and the word features of each word in the text, with the word features of each word serving as the local semantic features of the text; correspondingly, the above-mentioned matching of the text semantic features with each image global feature in the image information database to obtain a third image information list that matches the text semantic features may include the following steps: for any image global feature in the image information database, matching the image global feature with the global semantic features of the text and the local semantic features of the text respectively; determining the image information corresponding to the image global feature whose matching degree with the global semantic features of the text is greater than or equal to a first matching threshold and whose matching degree with the local semantic features of the text is greater than or equal to a second matching threshold as the third image information list.

[0061] For example, for any global feature of an image in the image database, the global feature of the image is matched with both the global semantic features and the local semantic features of the text. Considering that the local semantic features of the text include word features of multiple words in the search text, when matching the global feature of the image with the local semantic features of the text, it is necessary to match the global feature of the image with each of the multiple word features of the text.

[0062] Therefore, in this embodiment, for any global feature of an image, the matching result between the global feature of the image and the local semantic features of the text includes multiple matching degrees. In one implementation, the matching degree with the local semantic features of the text is greater than or equal to a second matching threshold. This can be that the matching degree corresponding to each word feature is greater than or equal to the second matching threshold, or the average value of the matching degrees corresponding to each word feature is greater than or equal to the second matching threshold. Of course, other situations are also possible, which will not be elaborated here.

[0063] In this embodiment of the application, when matching the text semantic features of the search text with the global features of the image, both the global semantic features and the local semantic features of the text are taken into account, which increases the dimension of feature matching. This makes the retrieval results not only match the overall semantics of the search text, but also match the semantics of the keywords in the search text, thereby improving the accuracy of the image retrieval results.

[0064] In one embodiment, step S201 above, obtaining keywords of the search text, may include the following steps: extracting search intent descriptors from the search text, the search intent descriptors including at least one of shooting time information, shooting location information, and object information contained in the image; obtaining target tags that match the search text from a tag library, the target tags being used to characterize object description information contained in the image; and determining the keywords based on the search intent descriptors and target tags.

[0065] To accurately extract search intent descriptors, categories of search intent descriptors can be predefined. For example, for the search intent descriptor "shooting time," a set of regular expression templates can be pre-defined to cover common time descriptions in Chinese. These templates are then used to match time-related text within the search results. For the search intent descriptor "shooting location," a geographical region table can be pre-obtained, and string matching can be used to find strings in the search text that match location names in the pre-obtained geographical region table, thus obtaining the aforementioned search intent descriptor.

[0066] Alternatively, in another embodiment, search intent description words can also be extracted from the search text based on a neural network model, such as a network model based on the Transformer architecture, a neural network based on the Long Short-Term Memory (LSTM) architecture, etc. Exemplarily, the types of search intent description words to be extracted can be predefined, and based on this, the neural network model can be trained to enable it to identify and extract search intent description words such as time information, location information, and object information contained in the picture from the search text. By adopting the method of using a neural network model, search intent description words can be extracted efficiently and accurately.

[0067] It should be noted that the above search intent description words refer to the vocabulary in the search text that describes the picture. However, in some embodiments, the search text input by the user may also contain description information about the objects in the picture. Therefore, in order to improve the accuracy of picture search, it is also possible to extract the vocabulary related to the object description information of the objects contained in the picture from the search text.

[0068] In one embodiment, a tag library can be pre-generated, and tags related to object description, such as "glasses", "yellow", "stripes", "sunset", etc., are stored in the tag library. Exemplarily, the target tags matching the search text can be obtained by matching the search text with the tag library. Additionally, it should be noted that for description information with the same semantics, there may be multiple corresponding tags stored in the tag library. For example, if the search text input by the user is "a photo of me wearing a dress taken yesterday", for the "dress" in this search text, there may be multiple different tags such as "one-piece dress", "dress", "skirt", "long dress", "short dress", etc. in the tag library.

[0069] In one embodiment, when obtaining the target tags by matching with the tag library, the search text can be first segmented into individual words, and meaningless words, such as stop words like "of", "is", etc., can be removed from the segmented words. The remaining words are then matched with the tag library to find the tags matching each word as the above target tags.

[0070] After obtaining the search intent description words of the search text and the above target tags, the above keywords are determined based on the search intent description words and target tags. In one embodiment, the search intent description words and target tags can be directly determined as the above keywords.

[0071] However, in some implementations, the words in the search text related to the description of the object contained in the image may or may not appear in the user's expected search terms. That is, the target tags matched in the above steps may or may not appear in the image the user expects. For example, if the search text is "a yellow cat photographed in Hangzhou yesterday," the target tag "yellow" is the tag the user expects to appear in the image. If the search text is "find people who don't wear glasses," the target tag "glasses" is the tag the user does not expect to appear in the image.

[0072] Therefore, to further improve the accuracy of image search, when determining keywords based on search intent descriptions and target tags, it's also possible to mark which target tags are expected and which are not. For example, one form of the keywords identified above could be: shooting time + shooting location + objects contained in the image + expected target tags + unexpected target tags.

[0073] Furthermore, user-input search text may contain unclear or non-standard descriptive terms. For example, a user might input "a yellow cat photographed in Hangzhou yesterday," where "yesterday" is the word indicating time. However, "yesterday" is not a clear descriptive term. Directly using "yesterday" in the image database may result in the inability to find images taken at the corresponding time or inaccurate results. Therefore, in this embodiment, when determining keywords based on search intent descriptions and target tags, it is necessary to convert search intent descriptions such as time and location. The converted search intent descriptions and the aforementioned target tags are then used to determine keywords. For example, time information is converted into a format representing year, month, day, hour, minute, and second. For instance, assuming yesterday was November 12, 2024, the search intent description "yesterday" in the above search text can be converted to "2024.11.1200:00:00-2024.11.12 23:59:59". Location information in search intent descriptions can also be converted into a standardized format, such as using the province, city, and district format. For example, "Hangzhou" can be converted to "Hangzhou City, Zhejiang Province," and "Yuhang" can be converted to "Yuhang District, Hangzhou City, Zhejiang Province, China."

[0074] In this embodiment, the search intent descriptive terms refer to descriptive words related to the image, and the target tags refer to descriptive words related to the objects in the image. Therefore, the obtained keywords consider both information related to the image description and the descriptive information of the objects contained in the image, ensuring the comprehensiveness of the extracted keywords and thus improving the accuracy of image search. Furthermore, given that the same feature of an object in an image may have multiple different expressions, a tag library matching method is used to determine the target tags used to describe the objects in the image. This ensures that diverse descriptions of objects can be captured, improving the comprehensiveness of the target tags and thus guaranteeing the comprehensiveness and accuracy of the search results.

[0075] In one implementation, the tag library stores pre-generated tag vectors for each tag. For example, the tag library can be pre-built, such as during the initialization phase of the image search system.

[0076] In this embodiment of the application, obtaining target tags that match the search text from the aforementioned tag library may include the following steps: for each word in the search text, searching for the corresponding word vector in the word vector library, which stores the word vectors corresponding to each word; for words for which no corresponding word vector can be found in the word vector library, calculating the corresponding word vector; searching for target tag vectors that match each word vector in the tag vector library; and generating corresponding target tags based on the target tag vectors.

[0077] It's easy to understand that since the tag library stores tag vectors for each tag, retrieving a target tag that matches the search text requires first calculating the word vectors for each word in the search text, and then finding the corresponding tag vectors through vector matching. However, considering that the word vectors for each word in the search text need to be calculated every time an image search is performed, this undoubtedly increases the computational load, thus affecting the image search speed.

[0078] Based on this, this embodiment of the application includes a word vector library, which pre-stores word vectors corresponding to a large number of frequently occurring words. Thus, when retrieving target tags matching the search text from the tag library, the search text is segmented to obtain the individual words contained within it. Then, the word vectors corresponding to each word in the search text are first searched in the word vector library. If a corresponding word vector can be found, it can be directly used to match the corresponding target tag vector from the tag library. For words whose corresponding word vectors cannot be found in the word vector library, the word vector corresponding to that word is calculated, and the corresponding target tag vector is matched from the tag library based on the calculated word vector.

[0079] To facilitate understanding, examples will be provided below.

[0080] For example, in one implementation, the search text is "search for a yellow cat photographed yesterday". First, the search text is segmented to obtain the constituent words: search / yesterday / photographed / of / yellow / of / cat. Exemplarily, before proceeding to subsequent steps, the words can be cleaned, such as removing meaningless words like "is" or "of" (stop words). Therefore, the cleaned words include search / yesterday / photographed / yellow / cat.

[0081] The algorithm searches the word vector library for the existence of word vectors corresponding to "search", "yesterday", "shoot", "yellow", and "cat". If word vectors for these words are found, they are used to search for corresponding target tag vectors in the tag library. If word vectors for "search", "yesterday", and "shoot" are found, but those for "yellow" and "cat" are not, word vectors for "yellow" and "cat" are calculated, and then these word vectors are used to search for corresponding target tag vectors in the tag library.

[0082] In this embodiment, by pre-storing a large number of word vectors corresponding to words in a word vector library, when performing image search based on search text, the word vectors corresponding to each word in the search text can be directly retrieved from the word vector library without the need to calculate these word vectors on-site. This processing scheme significantly reduces the computational burden in the process of obtaining target tags, thereby accelerating the retrieval speed of target tags and ultimately improving the efficiency of the entire image search.

[0083] Furthermore, considering that time or location information may appear as supplementary descriptions of an object in the search text, such as "Please search for a photo of the 20th-century oil painting I took," while "20th century" is a term representing time, it is not a search intent descriptor representing the shooting time in this search text. Using this term as the shooting time in the search would lead to no corresponding image being found or inaccurate search results. Therefore, when determining the keywords based on search intent descriptors and target tags, it is also necessary to determine whether the search intent descriptors represent the corresponding search intent. Therefore, in one implementation, determining the keywords based on search intent and target tags may include the following steps: combining search intent descriptors according to the information combination rules of a preset prompt template to obtain prompts; calling a large language model to identify whether the search intent descriptor has a target search intent based on the prompts; and determining the search intent descriptors with the target search intent and target tags as the keywords.

[0084] The information combination rules of the aforementioned preset prompt word template can be used to instruct the filling of search intent descriptors into the corresponding positions of the preset prompt word template. The preset prompt word template includes guiding words to guide the large language model in identifying the target search intent of the filled search intent descriptors. Thus, when the prompt word is input into the large language model, the model identifies whether the search intent descriptor in the prompt word has a target search intent according to the guidance of the prompt word. Different search intent descriptors have different search intents; therefore, the aforementioned target search intent representation is used to represent the search intent corresponding to the corresponding search intent descriptor. For example, if the search intent descriptor is the shooting time, the target search intent refers to searching for images taken at that time; if the search intent descriptor is the shooting location, the target search intent refers to searching for images taken at that location; if the search intent descriptor is an object contained in an image, the target search intent refers to searching for images containing that object.

[0085] If the above identification results indicate that a certain search intent descriptor does not have the target search intent, then the search intent descriptor will be removed and will not be included as a keyword.

[0086] In this embodiment of the application, when determining keywords based on search intent descriptors and target tags, a large language model is first used to identify whether each search intent descriptor has a target search intent. Then, search intent descriptors with target search intent and target tags are determined as the aforementioned keywords. This technical feature effectively eliminates the interference of descriptors without target search intent on image search results, thereby further improving the accuracy of image search.

[0087] In one implementation, the textual semantic features of the search text refer to the vector features of the search text, which typically require the use of a neural network model for vector feature extraction. Furthermore, in this embodiment, in addition to feature extraction of the search text, feature extraction of each image is also required to obtain the global image features corresponding to each image. Finally, the textual semantic features need to be matched with the global image features of each image. Therefore, it is necessary to ensure that the extracted textual semantic features and the global image features are within the same semantic space.

[0088] Therefore, based on the above considerations, a dual-tower model can be used for feature extraction of search text and images, so that the obtained text semantic features and image global features are in the same semantic space. Therefore, in this embodiment, the step S201 above, obtaining the text semantic features of the search text, can include the following steps: calling the dual-tower model, and extracting the text semantic features of the search text through the first tower model of the dual-tower model; wherein, the image global feature vector is obtained by extracting features from the image through the second tower model of the dual-tower model.

[0089] The dual-tower model is a deep learning model architecture whose core idea is to build independent feature representation networks (also called two towers) for two entities (such as text and images). This model architecture contains two parallel neural networks (one neural network is called one tower model), and each network consists of multiple layers of neural networks, specifically responsible for in-depth feature extraction from specific types of input data.

[0090] In this embodiment, one tower model of the dual-tower model processes the input search text. Through its multi-layer neural network architecture, it deeply mines and learns the intrinsic features of the search text, ultimately outputting a high-dimensional vector that represents the semantic information of the text, i.e., the text semantic features. The other tower model of the dual-tower model processes the input image. Through its multi-layer neural network architecture, it extracts features from the image, ultimately outputting a high-dimensional vector that represents the global features of the image, i.e., the global image features.

[0091] In one implementation, the search text is input into the first tower model of the dual-tower model, and the text feature extraction network in the first tower model extracts features from the search text to obtain text semantic features that can represent the core semantics of the search text.

[0092] For example, the text feature extraction network in the first tower model described above may include a word embedding layer, a feature extraction layer, and a pooling layer. The word embedding layer maps each word in the search text to a vector in a high-dimensional space, obtaining a text sequence represented by a high-dimensional vector. This text sequence is then input into the feature extraction network, which captures the sequence information and deep semantic features within the text sequence. For example, this feature extraction network may be composed of a recurrent neural network such as LSTM. The pooling layer performs pooling operations on the sequence information and deep semantic features output by the feature extraction network layer to extract key information, thereby generating the initial text features of the search text.

[0093] In one implementation, the image is input into the second tower model of the dual-tower model, and the image feature extraction network in the second tower model extracts features from the image to obtain global image features that can characterize the global features of the image.

[0094] For example, the image feature extraction network in the second tower model described above may include a convolutional neural network and pooling layers. The convolutional neural network captures visual information and local features in the image to extract features from the input image, while the pooling layers perform pooling operations on the features output by the convolutional neural network to extract key visual features, thereby generating the initial image features.

[0095] In this embodiment, feature extraction is performed on the search text and image through two branches of the dual-tower model, enabling the mapping of text semantic features and image global features into the same semantic space. This allows for accurate matching of the two, thereby improving matching precision. Furthermore, by using the dual-tower model for feature extraction, data from different modalities can be compared within the same semantic space, making cross-modal retrieval (such as image retrieval based on text) possible. Moreover, the dual-tower model directly obtains text semantic features and image global features within the same semantic space without requiring additional semantic space mapping operations, reducing computational complexity and resource consumption, and thus improving processing speed.

[0096] In step S203 above, the candidate images in the candidate image information set can be sorted based on their similarity. For example, they can be sorted in descending order of similarity. A preset number of candidate images can be selected as the image search results in descending order of similarity, or alternatively, candidate images with a similarity greater than or equal to a preset similarity threshold can be selected as the image search results.

[0097] The similarity score for any candidate image represents the similarity between the global features of the image and the semantic features of the text. For example, this similarity score can be calculated using a neural network model.

[0098] In one implementation, the similarity calculation can be performed using a neural network model architecture constructed from a feature fusion network and a similarity calculation network. For example, the feature fusion network can be a Transformer network, and the similarity calculation network can be a fully connected layer network.

[0099] In this embodiment, only the global feature vector of the image is used to calculate the similarity, without using the local feature vector. This reduces computational cost and performance overhead, while also improving computational speed and thus image search efficiency. However, considering that most existing models calculate similarity based on a combination of global and local image features, this embodiment requires designing and training a neural network model suitable for the scenario described in this application.

[0100] Therefore, in one embodiment, the method provided in this application further includes the following steps: inputting training samples into the model to be trained, wherein the training samples include a set of global image feature samples and text semantic feature samples; obtaining the predicted similarity output by the similarity calculation network, and calculating the loss function value based on the predicted similarity and the true similarity; when the loss function value reaches a preset convergence condition, determining the converged model to be trained as the similarity calculation model, wherein the similarity calculation model is used to calculate the similarity between the global image features and the text semantic features of each candidate image information in the candidate image information set.

[0101] Training the aforementioned model requires multiple training samples and their corresponding true similarity values. A training sample includes a set of global image feature samples and text semantic feature samples. During training, the training sample is input into the model, which predicts the similarity value between the global image feature samples and the text semantic feature samples within that training sample. Based on the predicted and true similarity values, the loss function of the model is calculated, and the parameters are adjusted accordingly. This process continues until the loss function reaches a preset convergence condition. When the loss function reaches the preset convergence condition, the current model is designated as the similarity calculation model, used to calculate the similarity between the global image features and the text semantic features of each candidate image in the candidate image information set.

[0102] For example, the aforementioned similarity calculation model can be composed of a feature fusion network and a similarity calculation grid. Specifically, the feature fusion network can be a Transformer network, and the similarity calculation network can be a fully connected layer network. When calculating the similarity between the global image features and the text semantic features of any candidate image information, the global image features and the text semantic features can be concatenated to obtain a concatenated feature. This concatenated feature is then input into the similarity calculation model. The self-attention mechanism in the Transformer network fuses the global image features and the text semantic features in the concatenated feature to obtain a fused feature vector. This fused feature vector is then input into the fully connected layer, which calculates the similarity value based on the fused feature vector.

[0103] In this embodiment, the similarity calculation model is trained based on global image features and semantic text features. This model can accurately determine the similarity between an image and the search text based on global image features and semantic text features, without needing to use local image features. This reduces computational overhead, increases computational speed, and thus improves image search efficiency. Furthermore, the image database only needs to store the global image features and image description information, without needing to store the image layout features, which greatly saves storage space.

[0104] In some application scenarios, user misoperation on the keyboard may cause the client to send an invalid image search request to the server. In this case, if the image retrieval method in this embodiment is still used to search for the image search request, it will lead to a waste of computing resources. Based on this, in this embodiment, before performing the above step S201, that is, before obtaining the text semantic features and keywords of the search text, the method provided in this embodiment further includes: extracting the lexical semantic features of each word in the search text; combining the lexical semantic features of each word; identifying whether the image search request is a valid search request based on the combined semantic features; if so, obtaining the text semantic features and keywords of the search text carried in the image search request.

[0105] For example, if the image search request is identified as not being a valid search request, that is, the image search request is invalid, then an empty value can be returned to the client as the image search result.

[0106] In one implementation, the semantic features of each word can be obtained based on the search text using a BERT model. These semantic features are then combined and input into a classifier, which classifies the search based on the combined features, outputting a "true" or "false" result. Here, "true" indicates a valid image search request, and "false" indicates an invalid one. In other implementations, "0" or "1" can also be used as the classifier's output.

[0107] For example, Figure 3 This is an exemplary computational flow diagram of the BERT model provided in this application embodiment. First, the search text input to the BERT model is segmented into words. The resulting word sequence is then input into the BERT model. The BERT model adds labels to the input word sequence to obtain word vectors for each word in the search text. Then, based on the word vectors, the semantic features of each word in the search text are obtained.

[0108] See Figure 3 For example, the search text is "Yesterday I took a picture of a yellow cat in Hangzhou". This search text is segmented into words, resulting in "yesterday", "in", "Hangzhou", "take", "of", "yellow", "of", and "cat". This word sequence is then input into the BERT model. The search text "Yesterday I took a picture of a yellow cat in Hangzhou" is a sentence. Therefore, the BERT model adds a label [CLS] to the word sequence "yesterday", "in", "Hangzhou", "take", "of", "yellow", "of", and "cat", and maps each word to a high-dimensional vector space using TokenEmbeddings. Position Embeddings identify the position of each word in the search text, thus representing the word vector and position vector of each word in the search text "Yesterday I took a picture of a yellow cat in Hangzhou" through these labels.

[0109] The [CLS] tag is used to indicate the first character of the search text. Token Embeddings are used to map each word to a high-dimensional vector space. Position Embeddings are used to identify the position of each word in the search text; for example, the word "yesterday" is at position "1", the word "at" is at position "2", and so on.

[0110] Furthermore, the Bert model obtains the lexical semantic features of each word in the text based on the aforementioned labels.

[0111] In this embodiment, before processing a received image search request, identifying whether the request is valid avoids processing invalid requests, preventing unnecessary resource waste. This allows computing resources to be concentrated on processing valid requests, leading to a more rational allocation and utilization of resources and ensuring efficient processing of valid search requests. Furthermore, by performing semantic recognition on the search text carried in the search request, it is possible to accurately determine whether the image search request is valid.

[0112] For example, Figure 4 This illustrates a second flowchart of the image search method provided in an embodiment of this application, as shown below. Figure 3 As shown, assuming the user inputs the search text "a yellow cat photographed in Hangzhou yesterday", the process first identifies whether the image search request is valid. If so, it further obtains the keywords of the search text and extracts the semantic features of the search text using a dual-tower model. Given the keywords, which include search intent descriptors and target tags, the system searches the image database for a first set of images matching the search intent descriptors and target tags, and a second set of images matching the semantic features and search intent descriptors. The first and second sets are then merged to obtain a candidate image set. The global image features of each image in the candidate set are concatenated with the aforementioned semantic features to obtain concatenated features. These concatenated features are then input into a similarity calculation model to calculate the similarity value between the global image features and the semantic features. Based on the similarity values, candidate images that meet the requirements are selected as the image retrieval results.

[0113] To enable image search using the aforementioned method, an image information database needs to be established. In one implementation, the global features and image description information of an image can be extracted and stored in the image information database simultaneously with the user's image storage. Therefore, the method provided in this application embodiment further includes the following steps: in response to an image storage request for an image to be stored, a dual-tower model is invoked to extract the global feature vector of the image to be stored through the second tower model in the dual-tower model; image detection is performed on the image to be stored to obtain the image description information of the image to be stored; and metadata extraction is performed on the image to be stored to obtain the image metadata of the image to be stored; the identification information of the image to be stored, the aforementioned global feature vector, the aforementioned image description information, and the image metadata are stored in the image information database.

[0114] The above-mentioned extraction of metadata from the images to be stored can be performed by extracting metadata from the images in the Exchangeable Image File (EXIF) format. The resulting image metadata may include the time and / or location of the image capture.

[0115] In one implementation, a user can send an image to be stored to a server via a client. Upon receiving the image storage request from the client, the storage system first stores the image in the user's designated storage space. Simultaneously, it detects the image to obtain its description, extracts metadata, and performs feature extraction to obtain global image features. Finally, it generates an image identifier, which is then stored in an image database along with the image description and global image features. For example, the image identifier, its corresponding description, metadata, and global image features can be stored in a table format.

[0116] In one implementation, if the objects contained in the image to be stored are people, object recognition can be performed on the objects contained in the image to be stored, and the images under user permissions can be clustered into groups, that is, images containing the same object can be grouped and labeled with an object identifier.

[0117] For example, the above-mentioned extraction of the global features of the image corresponding to the image to be stored can be performed by the second tower model in the dual-tower model. The specific extraction process has been described above and will not be repeated here.

[0118] In this embodiment of the application, the image to be stored can be detected by the Intelligent Media Management (IMM) system to obtain the corresponding image description information.

[0119] Figure 5 This document illustrates a flowchart of the construction process of the image information database provided in an embodiment of this application. Figure 5 As shown, for any image to be stored, the image is detected to obtain the corresponding image description information, and the metadata of the image to be stored is extracted to obtain the image metadata of the image to be stored. The global features of the image to be stored are extracted through the second tower model in the dual-tower model. The above-mentioned identification information, image description information, image metadata and global features of the image to be stored are stored in the table.

[0120] In this embodiment, whenever a user stores an image to be stored, in addition to storing the image, the global features and description information of the image to be stored are extracted, and image identification information of the image to be stored is generated and stored in the image information database in the form of a table. In this way, when a user needs to search for images, the search can be matched with relevant information in the image information database based on the user's input search text, which can realize fast and automated image search, improve the accuracy and efficiency of the search, and thus provide users with a more convenient and powerful image search service.

[0121] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide an image search device, such as... Figure 6 As shown, the image search device includes: an acquisition module 601, configured to acquire the text semantic features and keywords of the search text in response to receiving an image search request, wherein the search text is carried in the image search request and the keywords are words related to the search intent; a search module 602, configured to, upon acquiring the keywords, search the image information database for image information sets that match the text semantic features and the at least one keyword respectively, based on the image global features and image-related text description information of each image stored in the image information database, to obtain a candidate image information set; and a filtering module 603, configured to filter candidate image information that meets preset conditions as image search results according to the similarity between each candidate image information in the candidate image information set, wherein any similarity refers to the similarity between the image global features and the text semantic features of the corresponding candidate image information.

[0122] In one implementation, the keywords include search intent descriptors in the search text and target tags matching the search text. The search intent descriptors include at least one of shooting time information, shooting location information, and object information contained in the image. The target tags are used to characterize object description information of the objects contained in the image. The search module 602 is specifically used to: search the image information database for a first set of image information that matches the search intent descriptors and the target tags, and search the image information database for a second set of image information that matches the text semantic features and the search intent descriptors; merge the first set of image information and the second set of image information to obtain the candidate image information set.

[0123] In one embodiment, the image information database stores image-related text description information, including image description information and image metadata; the search module 602 is further configured to: match the search intent descriptor with each image metadata in the image information database to obtain a first image information list matching the search intent descriptor; match the target tag with each image description in the image information database to obtain a second image information list matching the target tag; and obtain a first image information set based on the image information common to the first image information list and the second image information list.

[0124] In one embodiment, the image information database stores image-related text description information including image metadata; the search module 602 is further specifically configured to: match the text semantic features with the global features of each image in the image information database to obtain a third image information list that matches the text semantic features; match the search intent descriptor with the image metadata of each image in the image information database to obtain a first image information list that matches the search intent descriptor; and obtain a second image information set based on the image information that coexists in the first image information list and the third image information list.

[0125] In one embodiment, the aforementioned text semantic features include global semantic features of the text and word features of each word in the text, wherein the word features of each word serve as local semantic features of the text; correspondingly, the aforementioned search module 602 is further specifically used to: for any global feature of an image in the image information database, match the global feature of the image with the global semantic features of the text and the local semantic features of the text respectively; and determine the image information corresponding to the global feature of the image that has a matching degree greater than or equal to a first matching threshold and a matching degree greater than or equal to a second matching threshold as the third image information list.

[0126] In one embodiment, the acquisition module 601 is specifically used to: extract search intent descriptors from the search text, the search intent descriptors including at least one of shooting time information, shooting location information, and object information contained in the image; obtain target tags matching the search text from a tag library, the target tags being used to characterize object description information of objects contained in the image; and determine the keywords based on the search intent descriptors and the target tags.

[0127] In one embodiment, the acquisition module 601 is further specifically used to: combine the search intent descriptive words according to the information combination rules of the preset prompt word template to obtain prompt words; call a large language model to identify whether the search intent descriptive words have a target search intent based on the prompt words; and determine the search intent descriptive words with the target search intent and the target tag as the keywords.

[0128] In one implementation, the tag library stores pre-generated tag vectors for each tag; the acquisition module 601 is further specifically configured to: for each word in the search text, search for the word vector corresponding to the word in the word vector library, the word vector library storing the word vectors corresponding to each word; for words for which no corresponding word vector can be found in the word vector library, calculate the word vector corresponding to the word; search for target tag vectors that match each word vector in the tag vector library; and generate corresponding target tags based on the target tag vectors.

[0129] In one embodiment, the acquisition module 601 is further configured to invoke a dual-tower model to extract the textual semantic features of the search text through the first tower model in the dual-tower model; wherein, the global feature vector of the image is obtained by extracting features from the image through the second tower model in the dual-tower model.

[0130] In one embodiment, the apparatus provided in this application further includes: an extraction module, used to extract the semantic features of each word in the search text; an identification module, used to combine the semantic features of each word and identify whether the image search request is a valid search request based on the combined features; and an acquisition module, used to acquire the text semantic features and keywords of the search text carried in the image search request when the identification module identifies the image search request as a valid search request.

[0131] In one embodiment, the apparatus provided in this application further includes: an input module for inputting training samples into a model to be trained, the training samples including a set of global image feature samples and text semantic feature samples; a calculation module for obtaining the predicted similarity output by the similarity calculation network and calculating a loss function value based on the predicted similarity and the true similarity; and a determination module for determining the converged model to be trained as a similarity calculation model when the loss function value reaches a preset convergence condition, the similarity calculation model being used to calculate the similarity between the global image features and the text semantic features of each candidate image information in the candidate image information set.

[0132] In one embodiment, the apparatus provided in this application further includes: a calling module, configured to, in response to an image storage request for an image to be stored, call a dual-tower model and extract the global feature vector of the image to be stored through the second tower model in the dual-tower model; a detection module, configured to perform image detection on the image to be stored to obtain image description information of the image to be stored, and to extract metadata from the image to be stored to obtain image metadata of the image to be stored; and a storage module, configured to store the identification information, the global feature vector, the image description information, and the image metadata of the image to be stored in the image information database.

[0133] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0134] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative, and the modules described as separate components may or may not be physically separate. The components illustrated as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0135] Understandable, Figure 6 The division of each module is merely a logical functional division. In actual implementation, the functions of these modules can be integrated into the server. For example, the functions of the first extraction module 601, the search module 602, and the filtering module 603 can be integrated into the server's processor.

[0136] Figure 7 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 7 As shown, the electronic device includes a memory 701 and a processor 702. The memory 701 stores a computer program that can run on the processor 702. When the processor 702 executes the computer program, it implements the method described in the above embodiments. The number of memories 701 and processors 702 can be one or more. In a specific implementation, the electronic device may also include a communication interface 703 for communicating with external devices and performing data exchange and transmission.

[0137] In practical implementation, if the memory 701, processor 702, and communication interface 703 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0138] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0139] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0140] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0141] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0142] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0143] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0144] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0145] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0146] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0148] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0149] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0150] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0151] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0152] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image search method, characterized in that, The method includes: In response to receiving an image search request, the semantic features and keywords of the search text are obtained, wherein the search text is carried in the image search request and the keywords are words related to the search intent; Once the keyword is obtained, based on the global image features of each image stored in the image information database and the text description information related to the image, a set of image information that matches the text semantic features and the keyword is searched from the image information database to obtain a set of candidate image information. Based on the similarity of each candidate image in the candidate image information set, candidate image information that meets preset conditions is selected as the image search result. Any similarity refers to the similarity between the global features of the corresponding candidate image information and the semantic features of the text.

2. The method according to claim 1, characterized in that, The keywords include search intent descriptors in the search text and target tags that match the search text. The search intent descriptors include at least one of shooting time information, shooting location information, and object information contained in the image. The target tags are used to characterize object description information of the objects contained in the image. The process involves searching the image information database for global image features and related textual descriptions of each image stored in the database, and then searching for image information sets that match the textual semantic features and keywords, respectively, to obtain a candidate image information set, including: The image information database is used to search for a first set of images that match the search intent descriptor and the target tag, and the image information database is used to search for a second set of images that match the text semantic features and the search intent descriptor. The first image information set and the second image information set are merged to obtain the candidate image information set.

3. The method according to claim 2, characterized in that, The image information database stores image-related text description information, including image description information and image metadata; the step of searching the image information database for a first set of image information that matches the search intent description and the target tag includes: The search intent descriptor is matched with the metadata of each image in the image information database to obtain a first image information list that matches the search intent descriptor. The target tag is matched with the description information of each image in the image information database to obtain a second image information list that matches the target tag; The first image information set is obtained based on the image information that exists in both the first image information list and the second image information list.

4. The method according to claim 2, characterized in that, The image information database stores image-related text description information, including image metadata; searching the image information database for a second set of image information that matches the text semantic features and the search intent description includes: The text semantic features are matched with the global features of each image in the image information database to obtain a third image information list that matches the text semantic features. The search intent descriptor is matched with the metadata of each image in the image information database to obtain a first image information list that matches the search intent descriptor. The second image information set is obtained based on the image information that exists in both the first image information list and the third image information list.

5. The method according to claim 4, characterized in that, The text semantic features include the global semantic features of the text and the word features of each word in the text, with the word features of each word serving as the local semantic features of the text; The step of matching the text semantic features with the global features of each image in the image information database to obtain a third image information list that matches the text semantic features includes: For any global feature of an image in the image information database, the global feature of the image is matched with the global semantic features and the local semantic features of the text, respectively. The image information corresponding to the global features of the image that match the global semantic features of the text with a degree greater than or equal to a first matching threshold and match the local semantic features of the text with a degree greater than or equal to a second matching threshold is determined as the third image information list.

6. The method according to any one of claims 1-5, characterized in that, Retrieve keywords from the search text, including: Extract search intent descriptors from the search text, wherein the search intent descriptors include at least one of shooting time information, shooting location information, and object information contained in the image; Obtain target tags that match the search text from the tag library; the target tags are used to characterize object description information of objects contained in the image. The keywords are determined based on the search intent descriptor and the target tag.

7. The method according to claim 6, characterized in that, Determining the keywords based on the search intent descriptor and the target tag includes: Based on the information combination rules of the preset prompt word template, the search intent descriptive words are combined to obtain prompt words; The large language model is invoked to identify whether the search intent descriptor has the target search intent based on the prompt words; The search intent descriptors with the target search intent and the target tag are determined as the keywords.

8. The method according to claim 6, characterized in that, The tag library stores the tag vectors of each pre-generated tag; The step of retrieving target tags that match the search text from the tag library includes: For each word in the search text, the word vector corresponding to the word is found from the word vector library, which stores the word vectors corresponding to each word. For words whose corresponding word vectors cannot be found in the word vector library, calculate the word vector corresponding to the word; Search the tag vector library for target tag vectors that match each of the word vectors; Generate the corresponding target label based on the target label vector.

9. The method according to any one of claims 1-5, characterized in that, Obtain the textual semantic features of the search text, including: The dual-tower model is invoked, and the textual semantic features of the search text are extracted through the first tower model in the dual-tower model; The global feature vector of the image is obtained by extracting features from the image using the second tower model in the dual-tower model.

10. The method according to any one of claims 1-5, characterized in that, Before obtaining the textual semantic features and keywords of the search text, the method further includes: Extract the lexical semantic features of each word in the search text; The semantic features of each word are combined, and the image search request is identified as a valid search request based on the combined features. If so, then obtain the text semantic features and keywords of the search text carried in the image search request.

11. The method according to any one of claims 1-5, characterized in that, The method further includes: The training samples are input into the model to be trained, and the training samples include a set of global image feature samples and text semantic feature samples; Obtain the predicted similarity output by the similarity calculation network, and calculate the loss function value based on the predicted similarity and the true similarity; When the loss function value reaches the preset convergence condition, the converged training model is determined as the similarity calculation model. The similarity calculation model is used to calculate the similarity between the global features of each candidate image in the candidate image information set and the semantic features of the text.

12. The method according to any one of claims 3-5, characterized in that, The method further includes: In response to an image storage request for an image to be stored, the dual-tower model is invoked, and the global feature vector of the image to be stored is extracted through the second tower model in the dual-tower model; The image to be stored is subjected to image detection to obtain the image description information of the image to be stored, and the image metadata of the image to be stored is extracted. The identification information of the image to be stored, the global feature vector, the image description information, and the image metadata are stored in the image information database.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 12.

14. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 12.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images