Image search method, system, device, medium and product

Abstract

The embodiment of the invention provides an image searching method, system and device, a medium and a product. For an original image serving as a keyword, the image searching method provided by the embodiment of the invention is used for searching images by images, namely finding images similar to the original image in alternative images. The image searching method comprises the following steps: determining a target image in alternative images according to a first similarity between the alternative images and an original image; and then taking an image title generated by the generative model for the target image as an image title of the original image, and determining a search result of the original image from the alternative images according to a second similarity between the image titles of the original image and the alternative images. Therefore, the method provides a new search logic, namely, the image search is realized by taking the image title in the text form as a basis. The image title can accurately describe the semantics of the image, so that the recall rate of image search can be ensured when the image title is used for image search.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image search method, system, device, medium and product. BACKGROUND

[0002] Image search is a technology that uses an image as a keyword for image retrieval, which allows users to search for other images similar or related to the uploaded picture from the database of candidate images.

[0003] In the image search scenario, the recall rate of an image refers to the proportion of relevant images that can be retrieved by the image search system to all relevant images. The recall rate can reflect the coverage ability of the system to the candidate images in the database, and the high or low of the recall rate can directly affect the search experience of the user.

[0004] Therefore, how to improve the recall rate of the image search result has become a problem to be solved. SUMMARY

[0005] Therefore, the embodiments of the present application provide an image search method, system, device, medium and product to improve the recall rate of the image search result.

[0006] The embodiments of the present application provide an image search method, which comprises: obtaining an original image as a keyword; determining a target image in the candidate images according to a first similarity between the candidate images and the original image; determining an image title of the target image as an image title of the original image by using a generative model; determining a search result corresponding to the original image in the candidate images according to a second similarity between the image titles of the candidate images and the original image.

[0007] The embodiments of the present application provide an image search system, which comprises: an online server and an offline server, and a generative model is deployed in the offline server; The online server is configured to receive an original image as a keyword sent by a user device; determining a target image in the candidate images according to a first similarity between the candidate images and the original image; querying an image title of the target image in the offline server to determine the image title of the target image as an image title of the original image; determining a search result corresponding to the original image in the candidate images according to a second similarity between the image titles of the candidate images and the original image; The offline server is configured to store the image title generated by the generative model for the target image.

[0008] An electronic device is provided, which includes a processor, a memory, and the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the image search method described above. The electronic device can further include a communication interface configured to communicate with other devices or communication networks.

[0009] An embodiment of the present application provides a non-transitory machine readable storage medium, and the non-transitory machine readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor can at least implement the image search method described above.

[0010] An embodiment of the present application provides a computer program product, and the computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the processor can implement the image search method described above.

[0011] For the original image as a keyword, the image search method provided by the embodiments of the present application is used to implement image search by image, that is, to find an image similar to the original image in the alternative image.

[0012] Therefore, the image search method provided by the embodiments of the present application can include: obtaining an original image of a keyword, and then determining a target image in the alternative image according to a first similarity between the alternative image and the original image. Then, an image title generated by a generative model for the target image is taken as an image title of the original image, and a search result corresponding to the original image is determined from the alternative image according to a second similarity between the image titles of the original image and the alternative image.

[0013] It can be seen that the above method provides a new search logic, that is, the image title in the form of text is used as a basis to implement image search by image. Since the image title can accurately describe the semantics of the image, the use of the image title for image search can ensure the recall rate of image search. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of an image search method provided by an embodiment of the present application; Figure 2 A flowchart of another image search method provided by an embodiment of the present application; Figure 3 a flow chart of another image search method provided by an embodiment of the present application; Figure 4 a working schematic diagram of an image search provided by an embodiment of the present application; Figure 5 a structural schematic diagram of an image search system provided by an embodiment of the present application; Figure 6 a structural schematic diagram of another image search system provided by an embodiment of the present application; Figure 7 a schematic diagram of a cloud computing environment provided by an embodiment of the present application; Figure 8 a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0016] The terms used in the embodiments of the present application are merely for the purpose of describing particular embodiments and are not intended to limit the present application. The singular forms “a”, “an” and “the” used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Plural” generally includes at least two, but does not exclude the case of including at least one.

[0017] It should be understood that the term “and / or” used herein merely describes an association relationship of associated objects, and means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character “ / ” herein generally represents an “or” relationship between the front and rear associated objects.

[0018] Depending on the context, the word “if” as used herein can be interpreted as meaning “when” or “upon” or “in response to determining” or “in response to identifying”. Similarly, depending on the context, the phrase “if it is determined” or “if it is identified (a stated condition or event)” can be interpreted as meaning “when it is determined” or “in response to determining” or “when it is identified (a stated condition or event)” or “in response to identifying (a stated condition or event)”.

[0019] It should be noted that, in the case that the embodiments of the present application involve user interaction operations or triggering operations, the user interaction operations or triggering operations involved in the embodiments of the present application include but are not limited to various modes of interaction operations such as touch operations, gesture operations, voice operations, head movement operations, eye movement operations, etc. The touch operation includes but is not limited to click operation, double-click operation, long-press operation, sliding operation, pinch operation or mouse hovering operation, etc. The sliding operation includes but is not limited to straight-line sliding, curve sliding, etc.

[0020] It should be noted that, in the case that the embodiments of the present application involve user information, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or rejection.

[0021] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such product or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the product or system including the element.

[0022] Before the embodiments provided by the present application are described in detail, the related concepts involved in the present application can also be introduced.

[0023] Image search by image: image search technology using images as search keywords, which allows users to search for similar or related images from the database of alternative images by uploading or specifying a picture.

[0024] Image search by image can be applied in e-commerce, content creation, online social networking and other scenarios. Specifically, in the e-commerce scenario, for example, a buyer can upload a picture of a clothing item he or she likes, and through image search, the same or similar style of goods can be returned. For example, a merchant can upload a picture of a product in his or her store, and through image search, counterfeit products existing on the e-commerce platform can be found. In the online social networking scenario, a user can upload a picture, and through image search, the user can search for social media content related to the uploaded image in the social media application program.

[0025] Generative model: A model built based on deep learning technology. Optionally, the generative model mentioned in the embodiments of this application can be a multimodal model. This model typically has an extremely large number of weights (ranging from billions to hundreds of billions), is capable of understanding data of at least one modality, and generates high-quality data. The data of at least one modality may include at least one of the following: files, audio, video, images, etc.

[0026] Optionally, generative models can be implemented based on neural network architectures, which can be obtained through pre-training on large amounts of data. In one optional implementation, the language model may include an encoder, a decoder, a self-attention layer, and a feed-forward neural network. The encoder converts the input data (usually in sequence form) into a vector representation, capturing the semantic features of the input data. The decoder converts the intermediate representations generated by the encoder into output data (usually in sequence form). The self-attention layer is a mechanism that allows the model to focus on other positions in the sequence to better encode information about the current position. The feed-forward neural network can perform non-linear transformations on the output of the self-attention layer to enhance the model's expressive power. The combined efforts of these components enable the language model built upon them to perform well in various complex tasks and possess strong generalization capabilities.

[0027] Furthermore, it should be noted that the embodiments of this application do not limit the number of model parameters supported by the generative models mentioned in each embodiment, with the goal of meeting actual needs. If there are relatively more model parameters, the model will be relatively larger in scale and have relatively better performance, but will consume more time and resources during inference or training. Conversely, if there are relatively fewer model parameters, the model will be relatively smaller in scale, and while meeting performance requirements, the model will be more lightweight, consuming relatively less time and resources during inference or training.

[0028] In image search scenarios, in order to improve the recall rate of image search, the methods and systems provided in the following embodiments of this application can be used.

[0029] The following detailed description of some embodiments of this application, in conjunction with the accompanying drawings, is provided. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0030] Figure 1This is a flowchart illustrating an image search method provided in an embodiment of this application. The method provided in this embodiment can be executed by an image search system. Furthermore, the image search system responds to image input in real time, outputting the search results corresponding to that image; that is, the image search system responds online to the input image. Figure 1 As shown, the method may include the following steps: S101, Obtain the original image as the keyword.

[0031] S102, Based on the first similarity between the candidate images and the original image, determine the target image from the candidate images.

[0032] Users can input the original image into the image search system (hereinafter referred to as the system in this embodiment and subsequent embodiments) using their user devices so that the system can obtain the original image.

[0033] The system can then determine the target image from the candidate images based on the initial similarity between the candidate images and the original image. The candidate images can be images acquired by the system before obtaining the original image; optionally, these candidate images can be images used as keywords to input "Xu Tao" before the original image is acquired, or they can be various images collected by the system from the internet. The target image can be an image with high similarity to the original image.

[0034] In the process of determining the target image based on similarity, in the first optional method, the system can calculate the first similarity between the candidate image and the original image, so that each candidate image can correspond to a first similarity. At this time, the system can determine the image corresponding to the highest first similarity as the target image, thereby obtaining a target image; the system can also determine the image corresponding to the first similarity greater than a first preset threshold as the target image, thereby obtaining at least one target image.

[0035] In this optional method, the system can calculate the similarity between the original image and each candidate image. Using this method, the target image with a high similarity to the original image can be more accurately identified.

[0036] In the second optional approach, the system can first cluster the candidate images to obtain at least one image cluster, where each cluster may contain a cluster center image. Then, the system can calculate a first similarity between each cluster center image and the original image. Specifically, the "first similarity between the candidate image and the original image" refers to the first similarity between the cluster center images in the candidate images and the original image. Each cluster center image corresponds to one first similarity. The system can then determine the cluster center image with the highest similarity or the cluster center image with a first similarity greater than a first preset threshold as the target image, thereby obtaining at least one target image.

[0037] In this alternative approach, the system can calculate similarity between the original image and at least one cluster center image. Since the number of cluster center images is smaller than the number of candidate images, this method can more quickly identify target images with high similarity to the original image. In other words, this method reduces the system's workload in calculating the first similarity by clustering the images, thereby improving the speed of target image identification.

[0038] Optionally, the first similarity between two images can be obtained using a pixel-based method, such as calculating the Euclidean distance, Manhattan distance, Hamming distance, or any other distance between corresponding pixels of the two images. This distance value can reflect the similarity between the images.

[0039] Alternatively, transformation-based methods can be used. For example, the Discrete Cosine Transform (DCT) can be used to transform the two images from the spatial domain to the frequency domain, and the magnitude of the coefficients in the frequency domain can reflect the similarity between the images. Alternatively, wavelet transform can be used to decompose the image into sub-band images of different scales and directions, and then the similarity between the features of the sub-band images of the two images can be calculated, and the similarity between the features of the sub-band images can be used as the similarity between the images.

[0040] Alternatively, feature-based methods can be used to obtain the similarity, such as calculating the similarity between the image features of two images and using this similarity as the image similarity. Image features can include at least one of texture features, color features, and shape features. Optionally, the similarity between image features can be represented by cosine similarity, and at least one of the aforementioned Euclidean distance, Manhattan distance, and Hamming distance. Furthermore, the image features can be extracted by the feature extraction model in the system.

[0041] When similarity is obtained using feature-based methods and the target image is determined using the second method described above, the process can be specifically described as follows: The feature extraction model in the system can extract first image features from the original image, and can also extract second image features from at least one cluster center image. Then, a first similarity between the first image features and the second image features is calculated. At this time, each second image feature corresponds to a first similarity. The image feature with the highest similarity or the first similarity greater than a first preset threshold can be determined as the target image feature. Finally, the image corresponding to the target image feature is determined as the target image.

[0042] S103, the image title generated by the generative model for the target image is determined as the image title of the original image.

[0043] S104, Based on the second similarity between the image titles of the candidate images and the original image, determine the search result corresponding to the original image among the candidate images.

[0044] The target image obtained in step S102 can be considered as an image among the candidate images that has a high similarity to the original image in terms of image details. Furthermore, the system can determine the image title generated by the generative model for the target image as the image title of the original image.

[0045] Ultimately, the system can determine the search result corresponding to the original image from among the candidate images based on the second similarity between the image titles of the candidate images and the original image. It's easy to understand that the image title is presented in text form. Optionally, each candidate image can correspond to a second similarity score, and the image corresponding to the second similarity score greater than a second preset threshold can be determined as the search result corresponding to the original image. The search result is the candidate image that has a high semantic similarity to the original image.

[0046] Optionally, the calculation of the second similarity between image titles and the determination of search results based on the second similarity can be implemented by a text search service deployed in the system. Furthermore, this embodiment does not limit the relationship between the first preset threshold in step S102 and the second preset threshold in this step.

[0047] In this embodiment, the original image of the keyword is obtained. Then, based on the first similarity between the candidate images and the original image, the target image is determined from the candidate images. The image title generated by the generative model for the target image is then used as the image title of the original image. Finally, based on the second similarity between the image titles of the original image and the candidate images, the search result corresponding to the original image is determined from the candidate images.

[0048] As can be seen, the above method provides a new search logic: using image titles in text form as the basis for image search. Since image titles can accurately describe the semantics of an image, using image titles for image search can guarantee the recall rate of image search.

[0049] exist Figure 1 Based on the illustrated embodiment, in order to improve the image search speed, Figure 2 A flowchart illustrating another image search method provided in an embodiment of this application. For example... Figure 2 As shown, the method may include the following steps: S201, Obtain the original image as the keyword.

[0050] The specific implementation of step S201 in this embodiment can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0051] S202, the first image features are extracted from the original image using a feature extraction model.

[0052] S203, Obtain the second image features, which include the clustering results of the image features of each candidate image.

[0053] The feature extraction model can extract features from both the original image and candidate images separately to obtain the image features of each image. Then, the system can cluster the image features of each candidate image to obtain at least one feature cluster, and each feature cluster can contain a cluster center feature. This at least one cluster center feature can constitute a second image feature. Optionally, the above clustering process to obtain the second image feature can be executed periodically by the system, rather than starting only in response to the reception of the original image. In other words, unlike the system's online response to the original image, the process of generating the second image feature is performed offline.

[0054] S204, determine the target image features based on the first similarity between the first image features and the second image features.

[0055] S205, determine the target image based on the image corresponding to the features of the target image.

[0056] Then, the system can determine the target image features based on the first similarity between the first image features and the second image features, and identify the image with the target image features as the target image. Optionally, as... Figure 1In the illustrated embodiment, the similarity between image features can be represented by cosine similarity, as well as any one of Euclidean distance, Manhattan distance, Hamming distance, etc. Furthermore, since the first image feature and the second image feature are extracted from the original image and the candidate image, respectively, the similarity between the first image feature and the second image feature can also be considered as the first similarity between the original image and the candidate image.

[0057] S206, The image title generated by the generative model for the target image is determined as the image title of the original image.

[0058] S207, Based on the second similarity between the image titles of the candidate images and the original image, determine the search result corresponding to the original image among the candidate images.

[0059] The specific implementation of steps S206 to S207 in this embodiment can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0060] In addition, for any content not described in detail in this embodiment, please refer to the descriptions in the relevant embodiments above, and will not be repeated here.

[0061] In this embodiment, after obtaining the original image, the system can extract the image features of both the original image and the candidate images, and obtain the second image features by clustering the image features of the candidate images. Then, the system can calculate a first similarity between the first and second image features to determine the target image features. Compared to the image features of all candidate images, the number of second image features is smaller; therefore, this embodiment can more quickly determine the target image with a high similarity to the original image. In other words, this embodiment reduces the system's workload in calculating the first similarity by clustering image features, thereby improving the speed of target image determination and further increasing the speed of image search. Simultaneously, the use of image titles can also improve the recall rate of search results.

[0062] To improve image search speed, Figure 3 A flowchart illustrating yet another image search method provided in an embodiment of this application. For example... Figure 3 As shown, the method may include the following steps: S301, Obtain the original image as the keyword.

[0063] S302, the first image features are extracted from the original image using a feature extraction model.

[0064] The specific implementation of steps S301 to S302 in this embodiment can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0065] S303, Obtain the second image features from the mapping table stored in the image search system. The second image features include the clustering results of the image features of each candidate image.

[0066] As Figure 2 As described in the illustrated embodiment, the system can cluster the image features of each candidate image to obtain a clustering result, i.e., at least one feature cluster. Each feature cluster can contain a cluster center feature, and the cluster center feature of each feature cluster can constitute a second image feature. In this embodiment, the second image feature can be stored in a mapping table in the system.

[0067] And just as Figure 2 As described in the illustrated embodiment, the process of clustering and obtaining the second image features can be performed periodically by the system, rather than after acquiring the original image. In other words, unlike the system's online response to the original image, the mapping table containing the second image features can be generated offline by the system; that is, the system can generate the mapping table periodically, rather than after acquiring the original image.

[0068] S304, determine the target image features based on the first similarity between the first image features and the second image features.

[0069] The specific implementation of step S304 in this embodiment can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0070] S305, based on the first mapping relationship between image features and cluster identifiers in the mapping table, query the target cluster identifier mapped by the target image features in the mapping table.

[0071] S306, Based on the second mapping relationship between cluster identifiers and image titles in the mapping table, query the target image title mapped by the target cluster identifier in the mapping table.

[0072] The mapping table generated offline by the system also includes a first mapping relationship between image features and cluster identifiers and a second mapping relationship between cluster identifiers and image titles. After obtaining the target image features, the system can obtain the target cluster identifier mapped by the target image features by querying the first mapping relationship in the mapping table. Furthermore, the system can obtain the target image title mapped by the target cluster identifier by querying the second mapping relationship in the mapping table.

[0073] S307, The image title generated from the target image is determined as the image title of the original image.

[0074] S308, Based on the second similarity between the image titles of the candidate images and the original image, determine the search result corresponding to the original image among the candidate images.

[0075] The specific implementation of steps S307 to S308 in this embodiment can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0076] In this embodiment, the system can respond online to the original image by utilizing a mapping table generated offline. Specifically, after obtaining the first image features of the original image, the system can obtain the image title of the original image by querying the mapping table. Then, the system can calculate the similarity between the image titles of candidate images and the original image, thereby finally outputting the search result corresponding to the original image. In other words, this embodiment can improve the speed of target image determination by looking up the table, thereby further improving the image search speed. Simultaneously, the use of image titles can also improve the recall rate of search results.

[0077] for Figures 2-3 The mapping table mentioned in the illustrated embodiment will be described in detail below, including its generation process: The system can acquire a first historical image, extract features from it, and then cluster the image features of the first historical image to obtain at least one feature cluster. Each feature cluster can also be assigned a cluster identifier. Simultaneously, the first historical image can also be input into a generative model within the system, which generates image titles for each image feature of the first historical image. The first historical image can include images input into the system before the original image within a first preset time period (e.g., 6 months). These first historical images can be images used as keywords input into the image search system within the first preset time period, or images collected by the system from the network.

[0078] Subsequently, for any feature cluster in at least one feature cluster, such as the first feature cluster, the system can create a first mapping relationship between the first cluster center feature of the first feature cluster and the cluster identifier of the first feature cluster; and create a second mapping relationship between the cluster identifier of the first feature cluster and the image title of the first cluster center feature. After creating the mapping relationship for each feature cluster in the above manner, the mapping table mentioned in the above embodiments can be obtained.

[0079] In this embodiment, the system can generate a mapping table based on the collected first historical images. Specifically, the mapping table may include the mapping relationship between image features, cluster identifiers, and image titles.

[0080] In practice, users can continuously input images as keywords into the system. Therefore, the historical images received by the system can be continuously updated, and consequently, the mapping table can also be continuously updated. The following is a detailed explanation of the mapping table update process: The system can acquire a second historical image, extract features from it, and then cluster the image features of the second historical image to obtain at least one feature cluster. Each feature cluster can also be assigned a cluster identifier. Simultaneously, the second historical image can also be input into a generative model within the system to generate image titles for each image feature of the second historical image.

[0081] It should be noted that the second historical image may include images input into the system within a second preset time period (e.g., 6 months) after the first historical image and before the original image. This second historical image may be an image input into the image search system as a keyword within the second preset time period, or it may be an image collected by the system from the network. Optionally, this embodiment does not limit the relationship between the second preset time period and the first preset time period. A common approach is that the first preset time period can be equal to the second preset time period, which may be equal to the generation cycle or update cycle of the mapping table.

[0082] Subsequently, for any feature cluster among at least one feature cluster obtained after clustering the image features of the second historical image, such as the second feature cluster, in one case, if the second cluster center feature of the second feature cluster is not included in the currently generated mapping table, the system can create a first mapping relationship between the second cluster center feature and the cluster identifier of the second feature cluster, and add the first mapping relationship to the currently generated mapping table; at the same time, the system can also create a second mapping relationship between the cluster identifier of the second feature cluster and the image title of the second cluster center feature, and add the second mapping relationship to the currently generated mapping table.

[0083] In another scenario, if the second cluster center feature of the second feature cluster is the same as any cluster center feature in the mapping table, such as the first cluster center feature, then the first mapping relationship between the first cluster center feature and the cluster identifier of the first feature cluster in the mapping table, as well as the second mapping relationship between the cluster identifier of the first feature cluster and the image title of the first cluster center image, can be maintained.

[0084] After updating the mapping relationship of each feature cluster in the above manner, the entire mapping table can be updated.

[0085] Optionally, the first historical image and the second historical image in this embodiment may be included in the alternative images.

[0086] In this embodiment, the system can update the mapping table based on the second historical image collected after the first historical image.

[0087] Figures 1-3In the illustrated embodiment, the system can determine the search results corresponding to the original image from the candidate images based on the second similarity between the image titles of the root candidate image and the original image. Optionally, the system can also determine the search results corresponding to the original image from the candidate images based on the first similarity between the candidate images and the original image. That is, the system can obtain a portion of the search results based on the first similarity and the second similarity, and these two portions of search results can be merged into the search results corresponding to the original image.

[0088] To improve the quality of search results, alternatively, duplicate search results can be processed from both parts of the search results, and the duplicate processing result can be used as the search result corresponding to the original image.

[0089] It should be noted that image features can describe image content, so using the first similarity is a search chain that determines the search results from the perspective of image content; image titles can describe image semantics, so using the second similarity is a search chain that determines the search results from the perspective of image semantics.

[0090] In this embodiment, the system can use two search links to comprehensively output the search results of the original image, that is, output the search results from the perspectives of image semantics and image content respectively, thereby improving the recall rate of search results.

[0091] Optionally, the image search system in the above embodiments may include an offline server and an online server. The offline server can create and store a mapping table; that is, the offline server can perform the above-described image feature clustering process to obtain second image features, and can also create a first mapping relationship between image features and cluster identifiers, and a second mapping relationship between cluster identifiers and image titles.

[0092] Offline servers can be deployed with feature extraction models to extract the individual image features of historical images. Offline servers can also be deployed with generative models, which can then generate the image titles in the mapping table.

[0093] The online server can receive the original image sent by the user's device, obtain the target image title by querying the mapping table on the offline server, and finally calculate and output the search results corresponding to the original image.

[0094] Based on the above description, the technical effects achievable by the embodiments of this application can also be understood in conjunction with the following: On the one hand, the image search system provided in the embodiments of this application is a system that combines online and offline servers. In this system, by leveraging the multimodal data processing capabilities of a generative model, the system can understand the semantics of the original image, i.e., generate an image title, thereby obtaining a search path that determines search results from an image semantic perspective, thus improving the recall rate of image searches.

[0095] Furthermore, offline servers process data offline, while online servers respond to input data in real time. In the embodiments of this application, the generative model is deployed on an offline server, meaning the process of generating image titles through generative model inference is offline. Compared to deploying the generative model on an online server, this saves computing resources on the online server, thereby improving the impact of the generative model's operation on image search speed.

[0096] On the other hand, in the various embodiments of this application, the process of generating image titles and the process of generating mapping tables by generative model inference are both offline. Therefore, image search can be achieved by using offline inference of offline server and online table lookup of online server, thereby improving the search efficiency of image search.

[0097] Therefore, the image search methods provided in the above embodiments of this application can also be combined with Figure 4 understand.

[0098] The following description, in conjunction with specific e-commerce scenarios, further illustrates the process of image search using the image search methods and systems provided in the various embodiments of this application: A user can open an e-commerce application installed on their device. This application's interface, such as the homepage, may include a search bar. The user can click the search bar and take a picture in the pop-up camera interface to use as the original image, or select an image stored on their device as the original image. This image taking or selection operation can be considered a search operation triggered by the user on their device, allowing the image search system to obtain the original image. Then, the image search can be performed using the methods described in the above embodiments. Finally, the search results can be displayed on the e-commerce application's search results page.

[0099] It should be noted that, although the executing entity of each step is specified in the description of the steps in the above embodiments of this application, this application does not limit the steps in the above method embodiments to the same device. That is, each step in the method embodiment can be executed by the same device, or the method can be executed by different devices. For example, the execution subject of steps S101 to S104 can be device A; or the execution subject of steps S101 and S102 can be device A, and the execution subject of steps S103 and S104 can be device B; the execution subject of steps S101 and S102 can be device A, the execution subject of step S103 can be device B, and the execution subject of step S104 can be device C, and so on.

[0100] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear in this document, or they may be executed in parallel. The operation numbers, such as S101, S102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0101] The above embodiments describe the operation of the image search system from a process perspective. Based on this, the operation process of the image system can also be described from a system structure perspective.

[0102] but Figure 5 This is a schematic diagram of the structure of an image search system provided in this embodiment, as shown below. Figure 5 As shown, the system may include an online server and an offline server, with a generative model also deployed on the offline server.

[0103] The online server receives original images sent by user devices as keywords and determines the target image from among the candidate images based on a first similarity score between the candidate images and the original image. The offline server queries the image title generated by the generative model for the target image to determine the target image's image title as the original image's image title. The process of determining the target image and its title is described above. Figures 1-3 The relevant descriptions in the illustrated embodiments will not be repeated here.

[0104] Finally, based on the second similarity between the image titles of the candidate images and the original image, the search result corresponding to the original image is determined from the candidate images. The specific calculation process for the first and second similarities is also described above. Figures 1-3 The relevant descriptions in the illustrated embodiments will not be repeated here.

[0105] In this embodiment, after obtaining the original image of the keyword, the online server can determine the target image from the candidate images based on the first similarity between the candidate images and the original image. Then, the image title generated by the generative model for the target image is used as the image title of the original image. Finally, based on the second similarity between the image titles of the original image and the candidate images, the search result corresponding to the original image is determined from the candidate images. The generative model is deployed on an offline server within the image search system, and the image title generated by the generative model for the target image is also stored on the offline server.

[0106] As can be seen, the above method provides a new search logic: the online server can use the text-based image titles obtained from the query and generated by the generative model on the offline server as the basis for image search. Since image titles can accurately describe the semantics of an image, using image titles for image search can guarantee the recall rate of image search.

[0107] Furthermore, any content not described in detail in this embodiment, as well as the technical effects that can be achieved, can be found in the descriptions of the above-mentioned method embodiments, and will not be repeated here.

[0108] Figure 6 This is a schematic diagram of another image search system provided in this embodiment. Optionally, as shown... Figure 6 As shown, the online server can also deploy feature extraction models.

[0109] The feature extraction model can extract features from the image input to the online server. Using the extracted image features, the online server can identify a target image that is highly similar to the original image from the candidate images.

[0110] Specifically, the online server can extract first image features from the original image using a feature extraction model. Then, it can retrieve second image features from a mapping table stored on the offline server. These second image features include the clustering results of the image features of each candidate image. Based on a first similarity between the first and second image features, the online server determines the target image feature and identifies the image corresponding to that target image feature as the target image.

[0111] Optionally, a feature extraction model can be deployed in the offline server, allowing the online server to obtain the target image features and their corresponding titles by querying the offline mapping table. Alternatively, the mapping table can also be generated and updated by the offline server; the query, generation, and updating processes for the mapping table are described in the relevant embodiments above and will not be repeated here.

[0112] In this embodiment, a mapping table can be obtained through image feature clustering. The online server can then perform image search by looking up the table, thereby improving the image search speed.

[0113] Furthermore, any content not described in detail in this embodiment, as well as the technical effects that can be achieved, can be found in the descriptions of the above-mentioned method embodiments, and will not be repeated here.

[0114] It should also be noted that the online server and the offline server with the generative model deployed in the image search system that executes the image search methods provided in the above embodiments of this application can both be server-side devices in a cloud computing environment. Optionally, the aforementioned server-side devices can be cloud servers maintained by cloud service providers—referred to as computing nodes. Client devices can be laptops, tablets, PCs, robots, etc.

[0115] In such Figure 7 The cloud computing environment shown may include several distributed deployments. Figure 7 The computing nodes (201-1, 201-2, ...) illustrated in the diagram can each possess processing resources such as computing and storage. In a cloud computing environment, multiple computing nodes can be organized to provide a certain service, such as the image search service mentioned in the embodiments of this application. Of course, a single computing node can also provide one or more services, such as... Figure 7 The diagram illustrates services A, B, C, and D. In a cloud computing environment, services can be provided through external service interfaces, which client devices can call to use the corresponding services. Service interfaces can take the form of Software Development Kits (SDKs) and Application Programming Interfaces (APIs).

[0116] The aforementioned services can be deployed using various virtualization technologies supported by the cloud computing environment, such as virtual machine-based and container-based virtualization technologies. Taking container-based virtualization technology as an example, several containers corresponding to a service can be assembled into a container group (pod). For example... Figure 7 The illustrated service B can be configured with one or more pods, and each pod can include a proxy and one or more containers. The one or more containers in the pod are used to handle requests related to one or more corresponding functions of the service, and the proxy in the pod is used to control network functions related to the service, such as routing and load balancing.

[0117] During operation, executing requests from client devices may require invoking one or more services in the cloud computing environment, and executing one or more functions of one service may require invoking one or more functions of another service. For example... Figure 7 As shown, after receiving a request from a client device, service A can call service B, and service B can request service D to perform one or more functions.

[0118] In one possible design, the image search methods provided in the above embodiments can be applied to electronic devices. For example... Figure 8 As shown, the electronic device may include a processor 21 and a memory 22. The memory 22 is used to store data that supports the electronic device in performing the above-described actions. Figures 1-4 The image search method program provided in the illustrated embodiment is configured by the processor 21 to execute the program stored in the memory 22.

[0119] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the processor 21, they can perform the following steps: Obtain the original image used as the keyword; The target image is determined from the candidate images based on a first similarity between the candidate images and the original image; The image title generated by the generative model for the target image is determined as the image title of the original image; Based on the second similarity between the image titles of the candidate images and the original image, the search result corresponding to the original image is determined among the candidate images.

[0120] Optionally, the processor 21 is further configured to: determine the search result corresponding to the original image in the candidate images based on the first similarity between the candidate images and the original image; The search results determined based on the first similarity and the search results determined based on the second similarity are deduplicated.

[0121] Optionally, when determining the target image from the candidate images based on the first similarity between the candidate images and the original image, the processor 21 is specifically configured to: extract a first image feature from the original image using a feature extraction model; Obtain a second image feature, which includes the clustering results of the image features of each of the candidate images; The target image features are determined based on the first similarity between the first image features and the second image features; The image corresponding to the target image features is determined as the target image.

[0122] Optionally, the electronic device is equipped with an image search system; When acquiring the second image feature, the processor 21 is specifically used to: acquire the second image feature from the mapping table stored in the image search system.

[0123] Optionally, when the processor 21 determines the image corresponding to the target image feature as the target image, it is specifically used to: query the target cluster identifier mapped by the target image feature in the mapping table according to the first mapping relationship between the image feature and the cluster identifier in the mapping table; Based on the second mapping relationship between cluster identifiers and image titles in the mapping table, the target image title mapped by the target cluster identifier is queried in the mapping table.

[0124] Optionally, the processor 21 is further configured to: cluster the image features of each of the first historical images to obtain at least one feature cluster, wherein the first historical images include images that serve as keywords within a first preset time period before the original image; A first mapping relationship is created between the first cluster center feature of the first feature cluster and the cluster identifier of the first feature cluster, wherein the first feature cluster includes any one of the at least one feature clusters; Create a second mapping relationship between the cluster identifier of the first feature cluster and the image title of the center feature of the first cluster.

[0125] Optionally, the processor 21 is further configured to: cluster the image features of each of the second historical images to obtain at least one feature cluster, wherein the second historical images include images that serve as keywords within a second preset time period after the first historical image and before the original image; If the second cluster center feature of the second feature cluster is not included in the mapping table, then the first mapping relationship between the second cluster center feature and the cluster identifier of the second feature cluster is added to the mapping table, and the second feature cluster includes any feature cluster among the at least one feature cluster; Add the first mapping relationship between the cluster identifier of the second feature cluster and the image title of the center image of the second cluster to the mapping table.

[0126] Optionally, the processor 21 is further configured to: input a cluster center image with cluster center features into the generative model, so that the generative model outputs an image title with the cluster center features. Wherein, the cluster center feature includes the first cluster center feature, and the cluster center image includes the first cluster center image, or the cluster center feature includes the second cluster center feature, and the cluster center image includes the second cluster center image.

[0127] Optionally, the processor 21 is further configured to: if the second cluster center feature of the second feature cluster is the same as the first cluster center feature in the mapping table, then maintain the first mapping relationship between the first cluster center feature and the cluster identifier of the first feature cluster in the mapping table, and the second mapping relationship between the cluster identifier of the first feature cluster and the image title of the first cluster center image.

[0128] The electronic device on which the image search system is deployed may include an offline server and an online server that communicates with the user device, wherein the offline server is deployed with the generative model; The original image is sent by the user equipment and received by the online server; The mapping table is generated and stored by the offline server, and the image titles in the mapping table are generated by the generative model. The title of the target image is obtained by querying the mapping table by the online server.

[0129] Optionally, the processor 21 is further configured to perform the aforementioned Figures 1-4 All or part of the steps in the illustrated embodiments.

[0130] The structure of the electronic device may also include other components such as a communication component 23, a display 24, a power supply component 25, and an audio component 26.

[0131] Figure 8 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 8 The components shown.

[0132] in addition, Figure 8 The components mentioned are optional, not mandatory, and depend on the specific product form of the electronic device. The electronic device in this application embodiment can be a conventional server, cloud server, or server array, etc.

[0133] The processor 21 can be any hardware processing device capable of executing the above-described method logic. Optionally, the processor can be a CPU, a graphics processing unit (GPU), or a microcontroller unit (MCU); it can also be a field-programmable gate array (FPGA), a programmable array logic (PAL), a general array logic (GAL), a complex programmable logic device (CPLD), or other programmable devices; or it can be an advanced RISC machine (ARM) or a system on chip (SoC), etc., but is not limited to these.

[0134] The aforementioned memory 22 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0135] The aforementioned communication component 23 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as Wi-Fi, 2G (e.g., Global System for Mobile Communications (GSM)), 3G (e.g., Wideband Code Division Multiple Access (WCDMA), 4G (e.g., Long Term Evolution (LTE)), 4G+ (e.g., LTE-Advanced (LTE-A)), or 5G (5th Generation Mobile Communication Technology), or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID), Infrared Data Association (IRDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0136] The aforementioned display 24 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0137] The aforementioned power supply component 25 provides power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0138] The audio component 26 described above can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0139] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), SRAM, dynamic random access memory (DRAM), other types of random-access memory (RAM), ROM, EEPROM, EPROM, PROM, flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.

[0140] Accordingly, this application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is able to implement the steps in the above method embodiments. These computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, so that the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device can be implemented as a means to implement the corresponding functions in the above method embodiments.

[0141] Furthermore, the specific implementation form of the computer program product is not limited in the embodiments of this application. In some embodiments, the computer program product may be implemented as an application (APP), a mini-program, a PC client, a program module, a plug-in, an installation package, a software development kit (SDK), an image file of an optical disc (such as an ISO file), a plug-in, or software in the form of Software as a Service (SaaS), etc., but is not limited to these.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image search method characterized by, The method comprises: obtaining an original image as a keyword; determining a target image in the candidate images according to a first similarity between the candidate images and the original image; determining an image title generated by a generative model for the target image as an image title of the original image; determining a search result corresponding to the original image in the candidate images according to a second similarity between the image titles of the candidate images and the original image.

2. The method of claim 1, wherein, The method further comprises: determining a search result corresponding to the original image in the candidate images according to the first similarity between the candidate images and the original image; de-duplicating the search result determined according to the first similarity and the search result determined according to the second similarity.

3. The method of claim 1, wherein, The determining of the target image in the candidate images according to the first similarity between the candidate images and the original image comprises: extracting a first image feature from the original image by using a feature extraction model; obtaining a second image feature, the second image feature comprising a clustering result of the respective image features of the candidate images; determining a target image feature according to a first similarity between the first image feature and the second image feature; determining an image corresponding to the target image feature as the target image.

4. The method of claim 3, wherein, The method is applied to an image search system; the obtaining of the second image feature comprises: obtaining the second image feature from a mapping table stored in the image search system.

5. The method of claim 3, wherein, The determining of the image corresponding to the target image feature as the target image comprises: querying a target cluster identifier mapped by the target image feature in the mapping table according to a first mapping relationship between the image features and the cluster identifiers in the mapping table; querying a target image title mapped by the target cluster identifier in the mapping table according to a second mapping relationship between the cluster identifiers and the image titles in the mapping table.

6. The method according to claim 3 or 4, characterized in that, The method further comprises: clustering the respective image features of first historical images to obtain at least one feature cluster, the first historical images comprising images as keywords before the original image within a first preset time period; creating a first mapping relationship between a first cluster center feature of a first feature cluster and a cluster identifier of the first feature cluster, the first feature cluster comprising any feature cluster in the at least one feature cluster; creating the second mapping relationship between the cluster identifier of the first feature cluster and an image title of the first cluster center feature.

7. The method of claim 6, wherein, The method further comprises: clustering the respective image features of second historical images to obtain at least one feature cluster, the second historical images comprising images as keywords after the first historical images within a second preset time period and before the original image; if a second cluster center feature of a second feature cluster is not contained in the mapping table, adding a first mapping relationship between the second cluster center feature and a cluster identifier of the second feature cluster to the mapping table, the second feature cluster comprising any feature cluster in the at least one feature cluster; add a first mapping relationship between a cluster identifier of the second feature cluster and an image title of the second cluster center image into the mapping table.

8. The method according to claim 6 or 7, characterized in that, The method further comprises: inputting a cluster center image with a cluster center feature into the generative model to output an image title of the cluster center feature by the generative model, wherein the cluster center feature comprises the first cluster center feature, the cluster center image comprises the first cluster center image, or the cluster center feature comprises a second cluster center feature, and the cluster center image comprises a second cluster center image.

9. The method of claim 7, wherein, The method further comprises: if a second cluster center feature of the second feature cluster is the same as the first cluster center feature in the mapping table, maintaining a first mapping relationship between the first cluster center feature and the cluster identifier of the first feature cluster in the mapping table, and the second mapping relationship between the cluster identifier of the first feature cluster and the image title of the first cluster center image.

10. The method of claim 5, wherein, The image search system comprises an offline server and an online server in communication with a user device, the offline server being deployed with the generative model; The original image is sent by the user device and received by the online server; The mapping table is generated and stored by the offline server, and the image title in the mapping table is generated by the generative model; The target image title of the target image is obtained by the online server querying the mapping table.

11. An image search system characterized by comprising: Comprise: An online server and an offline server, the offline server being deployed with a generative model; The online server receives an original image sent by a user device as a keyword; According to a first similarity between the alternative image and the original image, a target image is determined in the alternative image; An image title of the target image is queried in the offline server to determine the image title of the target image as the image title of the original image; According to a second similarity between the respective image titles of the alternative image and the original image, a search result corresponding to the original image is determined in the alternative image; The offline server stores the image title generated by the generative model for the target image.

12. The system of claim 11, wherein, The online server is deployed with a feature extraction model; The online server extracts a first image feature from the original image using the feature extraction model; A second image feature is obtained from a mapping table stored in the offline server, the second image feature comprising a clustering result of the respective image features of the alternative image; According to a first similarity between the first image feature and the second image feature, a target image feature is determined; The image corresponding to the target image feature is determined as the target image.

13. The system of claim 12, wherein The online server queries a target cluster identifier mapped by the target image feature in the mapping table according to a first mapping relationship between image features and cluster identifiers in the mapping table; The target image title mapped by the target cluster identifier is queried in the mapping table according to a second mapping relationship between cluster identifiers and image titles in the mapping table.

14. The system of claim 13, wherein, The offline server is configured to acquire a first historical image as a keyword before the original image; cluster respective image features of the first historical image to obtain at least one feature cluster; create a first mapping relationship between a first cluster center feature of a first feature cluster and a cluster identifier of the first feature cluster, the first feature cluster including any feature cluster in the at least one feature cluster; create the second mapping relationship between the cluster identifier of the first feature cluster and an image title of the first cluster center image.

15. The system of claim 14, wherein, The generative model is configured to receive a first cluster center image having the first cluster center feature; and output an image title of the first cluster center image.

16. An electronic device comprising: A memory and a processor; wherein the memory has executable code stored thereon, and when the executable code is executed by the processor, the processor performs the image search method according to any one of claims 1 to 10.

17. A non-transitory machine-readable storage medium, comprising: The non-transitory machine-readable storage medium has executable code stored thereon, and when the executable code is executed by the processor of the electronic device, the processor performs the image search method according to any one of claims 1 to 10.

18. A computer program product, characterised in that, The computer program product includes a computer program or instructions, which enable the computer program or instructions to implement the image search method according to any one of claims 1 to 10.

Images