Image search method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of deep learning technology, and discloses an image search method, device, computer equipment and storage medium, which adaptively clusters the image feature vector of each search image to obtain the cluster category vector of each search image, superimposes the image feature vector and the corresponding cluster category vector to obtain the fused feature vector of each search image, respectively calculates the first feature similarity of the search text vector and each fused feature vector, and determines the searched target image from the search image according to the numerical value of the first feature similarity. It can be seen that the present invention automatically finds the distinguishing features between the search images, that is, the cluster category vector, through adaptive clustering, so as to simultaneously search and match with the first text vector based on the image feature vector and the cluster category vector of the search image, and can achieve the purpose of improving the accuracy of image and text search compared to searching and matching based only on the image feature vector of the search image.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to an image search method, device, computer equipment and storage medium. Background Art

[0002] With the continuous development of science and technology, deep learning technology has been widely used in many technical fields, such as cross-modal image and text retrieval technology. Cross-modal image and text retrieval can realize mutual search between two different formats of data: text and pictures. That is, you can find the most suitable picture through text description, or you can search for the most suitable text description for the picture.

[0003] When performing cross-modal image-text retrieval, fine-grained image features of each image in the image database are usually extracted, and text features of the search text are extracted, and then the similarity between the text features of the search text and the fine-grained image features of each image is calculated, so that the image with the greatest similarity to the search text is selected as the final searched image.

[0004] However, this method is highly dependent on the segmentation granularity of the images in the base database. When there are images with the same label but different shapes, or images with different labels but the same shape in the base database, for example, the same plant has images with different shapes such as flowers, fruits, leaves, and branches, and images of different plants have similar flowers, fruits, leaves, and branches with the same shapes, if this phenomenon occurs in large numbers, the accuracy of cross-modal image and text retrieval will be reduced. Summary of the invention

[0005] Embodiments of the present invention provide an image search method, apparatus, computer device, and storage medium to solve the problem of reduced accuracy in cross-modal image and text retrieval.

[0006] An image search method, the method comprising: Obtaining a first text vector of the search text and an image feature vector of each search image in an image database; Adaptively clustering the image feature vectors to obtain a cluster category vector for each of the search images; Superimposing the image feature vector and the corresponding cluster category vector to obtain a fused feature vector of each search image; Calculating the first feature similarity of the search text vector and each of the fused feature vectors respectively; According to the numerical value of the first feature similarity, a searched target image is determined from the search images.

[0007] In the above method, optionally, determining the searched target image from the search image according to the numerical value of the first feature similarity includes: Determining a preset number of candidate images from the search image according to the numerical value of the first feature similarity; Acquire the image information text of each candidate image, and extract the second text vector of each image information text; Calculating the second feature similarity between the second text vector and the image feature vector of each candidate image and the first text vector; The candidate image corresponding to the second feature similarity with the largest value is determined as the target image.

[0008] In the above method, optionally, the calculating of the second feature similarity between the second text vector and the image feature vector of each candidate image and the first text vector comprises: Calculating the text similarity between the second text vector and the first text vector, and calculating the image-text similarity between the image feature vector and the first text vector; The second feature similarity is calculated based on the text similarity and the image-text similarity.

[0009] In the above method, optionally, the adaptive clustering of the image feature vector to obtain the cluster category vector of each search image includes: Adaptively clustering the image feature vector to obtain multiple image clustering categories; The vector average value is calculated for each of the image feature vectors corresponding to each of the image cluster categories to obtain a cluster category vector corresponding to each of the image feature vectors.

[0010] In the above method, optionally, the superposition of the image feature vector and the corresponding cluster category vector to obtain a fused feature vector of each search image includes: The corresponding cluster category vector is subtracted from each image feature vector to obtain a fused feature vector of each search image.

[0011] In the above method, optionally, the adaptive clustering of the image feature vector to obtain multiple image clustering categories includes: In the process of adaptively clustering the image feature vectors, the clustering centers of the image feature vectors are continuously adjusted; When the cluster centers meet the preset clustering condition, the image clustering category corresponding to each cluster center is determined.

[0012] In the above method, optionally, the step of obtaining the first text vector of the search text and the image feature vector of each search image in the image database includes: Acquire all search images in the image database; Performing feature extraction on each of the search images respectively to obtain the image feature vector of each of the search images; When the search text is acquired, feature extraction is performed on the search text to obtain the first text vector of the search text.

[0013] An image search device, comprising: A vector acquisition module, used to acquire a first text vector of the search text and an image feature vector of each search image in an image database; An adaptive clustering module, used for adaptively clustering the image feature vectors to obtain a clustering category vector for each search image; A feature fusion module, used for superimposing the image feature vector and the corresponding cluster category vector to obtain a fused feature vector of each search image; A similarity calculation module, used for respectively calculating the first feature similarity between the search text vector and each of the fused feature vectors; The image search module is used to determine the searched target image from the search image according to the numerical value of the first feature similarity.

[0014] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned image search methods when executing the computer program.

[0015] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements any of the above-mentioned image search methods.

[0016] The above-mentioned image search method, device, computer equipment and storage medium obtain the first text vector of the search text and the image feature vector of each search image in the image database, adaptively cluster the image feature vector, obtain the cluster category vector of each search image, superimpose the image feature vector and the corresponding cluster category vector to obtain the fused feature vector of each search image, respectively calculate the first feature similarity of the search text vector and each fused feature vector, and determine the searched target image from the search image according to the numerical value of the first feature similarity. It can be seen that the present invention automatically finds the distinguishing features between the search images, that is, the cluster category vector, through adaptive clustering, so as to simultaneously search and match with the first text vector based on the image feature vector and the cluster category vector of the search image. Compared with searching and matching based only on the image feature vector of the search image, the purpose of improving the accuracy of image and text search can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 is a flow chart for implementing an image search method disclosed in an embodiment of the present invention; Figure 2 It is a partial implementation flow chart of an image search method disclosed in one embodiment of the present invention; Figure 3 It is a partial implementation flow chart of an image search method disclosed in one embodiment of the present invention; Figure 4 It is a partial implementation flow chart of an image search method disclosed in one embodiment of the present invention; Figure 5 is another implementation flow chart of an image search method disclosed in an embodiment of the present invention; Figure 6 It is a partial implementation flow chart of an image search method disclosed in one embodiment of the present invention; Figure 7 It is a partial implementation flow chart of an image search method disclosed in one embodiment of the present invention; Figure 8 is a schematic diagram of the structure of an image search device disclosed in an embodiment of the present invention; Fig. 9 It is a structural schematic diagram of a computer device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0021] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] As used in the present specification and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0023] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0025] The present invention discloses an image search method, device, computer equipment and storage medium, which obtains the first text vector of the search text and the image feature vector of each search image in the image database, adaptively clusters the image feature vector, obtains the cluster category vector of each search image, superimposes the image feature vector and the corresponding cluster category vector, obtains the fused feature vector of each search image, respectively calculates the first feature similarity of the search text vector and each fused feature vector, and determines the searched target image from the search image according to the numerical value of the first feature similarity. It can be seen that the present invention obtains the fused feature vector by fusing the image feature vector and the cluster category vector of the search image, and then calculates the first feature similarity of the first text vector of the search text and each fused feature vector, and determines the target image from the search image according to the numerical value of the first feature similarity. Compared with the prior art, it is beneficial to improve the search accuracy of cross-modal image-text retrieval. It is explained below through specific embodiments.

[0026] like Figure 1 FIG. 1 is a flowchart of an implementation of an image search method disclosed in an embodiment of the present invention. The method is applicable to any electronic device with image processing capability, such as a mobile phone, a tablet computer, a notebook computer, a personal computer or a server. The method in this embodiment may specifically include the following steps: S101: Obtain a first text vector of a search text and an image feature vector of each search image in an image database.

[0027] It should be understood that a large number of search images are stored in the image database, and to search the search images stored in the image database based on the search text, it is necessary to perform feature extraction on all the search images in the image database to obtain the image feature vector of each search image, and perform feature extraction on the search text to obtain the first text vector. The image feature vector of each search image is a fine-grained feature vector of the search image. In a specific implementation, in this embodiment, a pre-trained feature extractor can be used to perform feature extraction on each search image to obtain image features of each search image, and the extracted image features are converted into image feature vectors, wherein the feature extractor in this embodiment includes but is not limited to training one or more models of a VIT model and a CNN model.

[0028] In addition, in this embodiment, the search text can be feature extracted by a text feature extraction model to obtain text features, and the extracted text features are converted into text feature vectors, wherein the text feature extraction model includes but is not limited to any one of the BERT model and the TF-IDF model.

[0029] It should be noted that the above are only part of the implementation methods of feature extraction for search images and search texts in this embodiment, and the specific implementation methods of feature extraction for search images and search texts are not limited in this embodiment.

[0030] S102: Adaptively clustering the image feature vectors to obtain a clustering category vector for each search image.

[0031] In a specific implementation, the clustering method for adaptively clustering image feature vectors in this embodiment includes but is not limited to any one of the adaptive clustering algorithms such as the InfoMap adaptive clustering algorithm, the density-based adaptive clustering algorithm, and the Mean Shift adaptive clustering algorithm. When the image feature vector of each search image is extracted from the image database, adaptive clustering is performed based on the image feature vector of each search image to obtain multiple cluster centers and the image feature vector corresponding to each cluster center. The cluster category vector of each search image is determined based on the cluster center, and the cluster category vectors of the search images with the same cluster center are the same.

[0032] S103: Superimposing the image feature vector and the corresponding cluster category vector to obtain a fused feature vector of each search image.

[0033] In a specific implementation, this embodiment includes but is not limited to superimposing the image feature vector and the corresponding cluster category vector by any one of the methods such as average fusion, weighted average fusion, and splicing fusion.

[0034] S104: Calculate the first feature similarity between the search text vector and each fused feature vector respectively.

[0035] In a specific implementation, in this embodiment, any one of the cosine similarity, Euclidean distance, Manhattan distance, etc. between the search text vector and each fused feature vector can be calculated as the first feature similarity between the search text vector and each fused feature vector. In this embodiment, no specific limitation is made to the calculated first feature similarity.

[0036] S105: Determine the searched target image from the search images according to the numerical value of the first feature similarity.

[0037] It should be understood that the larger the value of the first feature similarity is, the higher the matching degree between the corresponding search image and the search text is, and the smaller the value of the first feature similarity is, the lower the matching degree between the corresponding search image and the search text is. Therefore, the search image corresponding to the first feature similarity with the largest value can be selected as the searched target image, or multiple corresponding search images can be selected as the searched target images in order of the first feature display degree values ​​from large to small.

[0038] In summary, the present invention discloses an image search method, which obtains the first text vector of the search text and the image feature vector of each search image in the image database, adaptively clusters the image feature vector, obtains the cluster category vector of each search image, superimposes the image feature vector and the corresponding cluster category vector, obtains the fused feature vector of each search image, respectively calculates the first feature similarity of the search text vector and each fused feature vector, and determines the searched target image from the search image according to the numerical value of the first feature similarity. It can be seen that the present invention automatically finds the distinguishing features between the search images, namely the cluster category vector, through adaptive clustering, so as to simultaneously search and match with the first text vector based on the image feature vector and the cluster category vector of the search image. Compared with searching and matching based only on the image feature vector of the search image, the purpose of improving the accuracy of image and text search can be achieved.

[0039] based on Figure 1 In the specific implementation, step S105 in this embodiment can be implemented by the following steps: Figure 2 As shown: S201: Determine a preset number of candidate images from the search image according to the numerical value of the first feature similarity.

[0040] According to the value of the first feature similarity, the search images are sorted from large to small, and then a preset number of search images are selected from large to small as candidate images. The preset number can be set according to actual needs, such as 10, 20 or 30, etc., and the specific value of the preset number is not limited in this embodiment.

[0041] S202: Obtain image information text of each candidate image, and extract a second text vector of each image information text.

[0042] It should be noted that each search image in this embodiment corresponds to an image information text, which is used to provide a detailed introduction to the content in the search image. Therefore, in determining the alternative images from the search image, the image information text corresponding to each alternative image can be obtained, and the second text vector of each image information text can be extracted.

[0043] For example, taking the content of the search image as Artemisia selengensis, the detailed introduction of Artemisia selengensis can be "It is a perennial herb of the genus Artemisia in the Asteraceae family. The stem is upright or obliquely upward, with strong branching, the young stem is purple-red, and white creeping rhizomes grow underground. It is mainly distributed in Jiangsu, Anhui, Hubei, Jiangxi, Yunnan, Northeast China and other places in China." Therefore, by obtaining the detailed introduction of Artemisia selengensis, the image information text of Artemisia selengensis can be obtained, and then the second text vector can be extracted from the image information text.

[0044] S203: Calculate the second feature similarity between the second text vector and the image feature vector of each candidate image and the first text vector.

[0045] That is, after obtaining the second text vector and image feature vector of each candidate image, the similarity between the second text vector and image feature vector of the candidate image and the second feature of the first text vector is calculated.

[0046] In a specific implementation, the refined ranking model in this embodiment can be obtained based on deep neural network (DNN) training. The second text vector and image feature vector of the candidate image are fitted by the refined ranking model, and the second feature similarity with the first text vector is a fractional value between 0 and 1.

[0047] S204: Determine the candidate image corresponding to the second feature similarity with the largest value as the target image.

[0048] In a specific implementation, in this embodiment, the candidate images may be sorted according to the numerical values ​​of the second feature similarities, and the candidate image corresponding to the second feature similarity with the largest numerical value is selected as the final target image.

[0049] To sum up, in this embodiment, a preset number of candidate images are screened out from the search image, and then the candidate images are further screened through the fine ranking model based on the second text vector and image feature vector of each candidate image and the first text vector. This not only significantly improves the accuracy of image and text search, but also avoids the huge pressure brought by directly using the fine ranking model to sort the search images, which is conducive to improving the efficiency of image and text search.

[0050] based on Figure 2 In the specific implementation, step S104 in this embodiment can also be implemented by the following steps: Figure 3 As shown: S301: Calculate the text similarity between the second text vector and the first text vector, and calculate the image-text similarity between the image feature vector and the first text vector.

[0051] S302: Based on the text similarity and the image-text similarity, a second feature similarity is calculated.

[0052] In a specific implementation, in this embodiment, the text similarity between the second text vector and the first text vector can be calculated by calculating any one of the cosine similarity, Euclidean distance, Manhattan distance, etc. between the second text vector and the first text vector, and the image feature vector and the first text vector can be calculated by calculating any one of the cosine similarity, Euclidean distance, Manhattan distance, etc. between the image feature vector and the first text vector as the image-text similarity between the image feature vector and the first text vector. In this embodiment, the calculated text similarity and image-text similarity are not specifically limited.

[0053] In addition, in this embodiment, after the text similarity and the image-text similarity are calculated, the vector average value or weighted vector average value of the text similarity and the image-text similarity can be calculated as the second feature similarity.

[0054] To sum up, in this embodiment, the corresponding second feature similarity is calculated through the joint action of text similarity and image-text similarity, and then the target image is searched from the alternative images based on the second feature similarity. Compared with the target image searched based only on image-text similarity, the search accuracy can be further improved.

[0055] based on Figure 1 In the specific implementation, step S102 in this embodiment can be implemented by the following steps: Figure 4 As shown: S401: Adaptively clustering the image feature vector to obtain multiple image clustering categories.

[0056] After the corresponding image feature vector is extracted from the search image, the image feature vector is adaptively clustered by an adaptive clustering algorithm to obtain multiple cluster centers, and the image feature vector in each cluster center corresponds to a same image cluster category.

[0057] In a specific implementation, the clustering method for adaptively clustering image feature vectors in this embodiment includes but is not limited to any one of the adaptive clustering algorithms such as the InfoMap adaptive clustering algorithm, the density-based adaptive clustering algorithm, and the Mean Shift adaptive clustering algorithm. When the image feature vector of each search image is extracted from the image database, adaptive clustering is performed based on the image feature vector of each search image to obtain multiple cluster centers and the image feature vector corresponding to each cluster center. The cluster category vector of each search image is determined based on the cluster center, and the cluster category vectors of the search images with the same cluster center are the same.

[0058] S402: Calculate the vector average value of the image feature vectors corresponding to each image cluster category to obtain the cluster category vector corresponding to each image feature vector.

[0059] In a specific implementation, in this embodiment, the image feature vector corresponding to each image clustering category, that is, the image feature vector corresponding to each cluster center, is calculated as a vector average. The obtained average vector is the cluster category vector corresponding to each search image in the cluster center. Thus, the cluster category vector and image feature vector corresponding to each search image can be obtained.

[0060] To sum up, in this embodiment, the vector average of the image feature vectors of each search image corresponding to each cluster center is calculated to obtain the average vector, and then the average vector is used as the cluster category vector of each search image corresponding to the cluster center, thereby avoiding the impact of reconstructing the cluster category vector of each search image on the accuracy of image and text search.

[0061] like Figure 1As shown, step S103 in this embodiment can also be implemented by the following steps: Figure 5 As shown: S106: Subtract the corresponding cluster category vector from each image feature vector to obtain a fused feature vector of each search image.

[0062] In a specific implementation, in this embodiment, the image feature vector and the cluster category vector of each search image can be subtracted to obtain a fused feature vector. The fused feature vector can be the image feature vector minus the corresponding cluster category vector, or the cluster category vector minus the corresponding image feature vector, which is not limited in this embodiment.

[0063] based on Figure 4 In the specific implementation, step S401 in this embodiment can be implemented by the following steps: Figure 6 As shown: S601: In the process of adaptively clustering the image feature vectors, the clustering centers of the image feature vectors are continuously adjusted.

[0064] After extracting the image feature vector from each search image, the image feature vector is adaptively clustered to obtain multiple cluster centers of the image feature vector, and then it is determined whether the cluster center meets the preset clustering condition. If the cluster center does not meet the preset clustering condition, the image feature vector is re-extracted from the search image, and it is determined again whether the cluster center meets the preset clustering condition, and so on, until the cluster center meets the preset clustering condition, and the image clustering category corresponding to each cluster center is determined.

[0065] S602: When the cluster center reaches a preset clustering condition, a plurality of image clustering categories are obtained.

[0066] In a specific implementation, the clustering conditions in this embodiment are as follows: First, the clustering condition in this embodiment can be to reach a preset number of clustering times, that is, when the image feature vector is extracted from the search image and the number of clustering of the image feature vector reaches the preset number of clustering times, the corresponding image clustering category of each cluster center is determined, that is, each cluster center corresponds to an image clustering category.

[0067] Secondly, the clustering condition in this embodiment may be that when the cluster center no longer changes, the image cluster category corresponding to each cluster center is determined, that is, each cluster center corresponds to an image cluster category.

[0068] In summary, in this embodiment, the image clustering categories are obtained by the adaptive clustering method, which avoids the influence of human intervention on the clustering results, can make the obtained image clustering categories more accurate, and is conducive to improving the search accuracy of the final image and text search.

[0069] based on Figure 1 In the specific implementation, step S101 in this embodiment can be implemented by the following steps: Figure 7 As shown: S701: Acquire all search images in the image database.

[0070] It is understandable that a large number of search images are stored in the image database. When searching the search images in the image database, each search image needs to be matched separately to finally search for the desired image. Therefore, all search images in the image database need to be obtained.

[0071] S702: Perform feature extraction on each search image to obtain an image feature vector for each search image.

[0072] In one implementation, in this embodiment, a feature database corresponding to an image database can be established, and feature extraction can be performed on all search images in the image database. After obtaining the image feature vector of each search image, the image feature vector can be stored in the feature database. Then, when searching for the search image in the image database, the image feature vector can be directly obtained from the feature database.

[0073] In a specific implementation, in this embodiment, a pre-trained feature extractor can be used to perform feature extraction on each search image to obtain image features of each search image, and the extracted image features are converted into image feature vectors, wherein the feature extractor in this embodiment includes but is not limited to training one or more models of a VIT model and a CNN model.

[0074] S703: When the search text is obtained, feature extraction is performed on the search text to obtain a first text vector of the search text.

[0075] It should be understood that in this embodiment, feature extraction can be performed on the search image in the image database at any time. Therefore, feature extraction can be performed on the search image in the image database in advance, and then when the search text is obtained, the search text is feature extracted to obtain the first text vector, and then the image feature vector can be searched according to the first text vector.

[0076] Specifically, in this embodiment, the search text may be subjected to feature extraction through a text feature extraction model to obtain text features, and the extracted text features may be converted into text feature vectors, wherein the text feature extraction model includes but is not limited to any one of a BERT model and a TF-IDF model.

[0077] To sum up, in this embodiment, by pre-extracting features from each search image to obtain an image feature vector for each search image, and storing the image feature vector in a feature database, when the search text is obtained, the image feature vector can be searched immediately based on the first text vector of the search text, which is beneficial to improving the search efficiency of the image and text search in this embodiment.

[0078] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0079] like Figure 8 The figure is a schematic diagram of the structure of an image search device disclosed in the present invention, which is applicable to electronic devices with image processing capabilities, such as any device in a mobile phone, a tablet computer, a notebook computer, a personal computer or a server. The device in this embodiment may specifically include the following modules: A vector acquisition module 801, used to acquire a first text vector of the search text and an image feature vector of each search image in the image database; An adaptive clustering module 802 is used to adaptively cluster the image feature vectors to obtain a clustering category vector for each search image; A feature fusion module 803 is used to superimpose the image feature vector and the corresponding cluster category vector to obtain a fused feature vector of each search image; A similarity calculation module 804 is used to calculate the first feature similarity of the search text vector and each fused feature vector respectively; The image search module 805 is used to determine the searched target image from the search images according to the numerical value of the first feature similarity.

[0080] In summary, an image search device disclosed in the present embodiment obtains the first text vector of the search text and the image feature vector of each search image in the image database, adaptively clusters the image feature vector, obtains the cluster category vector of each search image, superimposes the image feature vector and the corresponding cluster category vector, obtains the fused feature vector of each search image, respectively calculates the first feature similarity of the search text vector and each fused feature vector, and determines the searched target image from the search image according to the numerical value of the first feature similarity. It can be seen that the present invention automatically finds the distinguishing features between the search images, namely the cluster category vector, through adaptive clustering, so as to simultaneously search and match with the first text vector based on the image feature vector and the cluster category vector of the search image. Compared with searching and matching based only on the image feature vector of the search image, the purpose of improving the accuracy of image and text search can be achieved.

[0081] In one embodiment, the image search module 805 may be used to: Determining a preset number of candidate images from the search image according to the numerical value of the first feature similarity; Obtaining image information text of each candidate image, and extracting a second text vector of each image information text; Calculate the similarity between the second text vector and the image feature vector of each candidate image and the second feature of the first text vector; The candidate image corresponding to the second feature similarity with the largest value is determined as the target image.

[0082] In one embodiment, the image search module 805 may also be used to: Calculating the text similarity between the second text vector and the first text vector, and calculating the image-text similarity between the image feature vector and the first text vector; Based on the text similarity and the image-text similarity, the second feature similarity is calculated.

[0083] In one embodiment, the adaptive clustering module 802 may be used to: Adaptively cluster the image feature vectors to obtain multiple image clustering categories; The vector average value of the image feature vector corresponding to each image clustering category is calculated respectively to obtain the clustering category vector corresponding to each image feature vector.

[0084] In one embodiment, the adaptive clustering module 802 may also be used to: Subtract the corresponding cluster category vector from each image feature vector to obtain the fused feature vector of each search image.

[0085] In one embodiment, the adaptive clustering module 802 may also be used to: In the process of adaptively clustering the image feature vector, the cluster center of the image feature vector is continuously adjusted; When the cluster centers meet the preset clustering conditions, the image clustering category corresponding to each cluster center is determined.

[0086] In one embodiment, the vector acquisition module 801 in this embodiment can be used to: Get all search images in the image database; Perform feature extraction on each search image respectively to obtain an image feature vector of each search image; When the search text is obtained, feature extraction is performed on the search text to obtain a first text vector of the search text.

[0087] For the specific definition of the image search device, please refer to the relevant definition of the image search method above, which will not be repeated here. Each module in the above-mentioned image search device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0088] The embodiment of the present application also discloses a computer device, which may be a server, and its internal structure diagram may be as shown in Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image search method is implemented.

[0089] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Obtaining a first text vector of the search text and an image feature vector of each search image in an image database; Adaptively cluster the image feature vectors to obtain the cluster category vector of each search image; Superimpose the image feature vector and the corresponding cluster category vector to obtain the fused feature vector of each search image; Calculate the similarity of the first feature of the search text vector and each fused feature vector respectively; According to the numerical value of the first feature similarity, a searched target image is determined from the search images.

[0090] In one embodiment, determining the searched target image from the search image according to the numerical value of the first feature similarity includes: Determining a preset number of candidate images from the search image according to the numerical value of the first feature similarity; Obtaining image information text of each candidate image, and extracting a second text vector of each image information text; Calculate the similarity between the second text vector and the image feature vector of each candidate image and the second feature of the first text vector; The candidate image corresponding to the second feature similarity with the largest value is determined as the target image.

[0091] In one embodiment, calculating the similarity between the second text vector and the image feature vector of each candidate image and the second feature of the first text vector includes: Calculating the text similarity between the second text vector and the first text vector, and calculating the image-text similarity between the image feature vector and the first text vector; Based on the text similarity and the image-text similarity, the second feature similarity is calculated.

[0092] In one embodiment, adaptive clustering is performed on the image feature vector to obtain a cluster category vector for each search image, including: Adaptively cluster the image feature vectors to obtain multiple image clustering categories; The vector average value of the image feature vector corresponding to each image clustering category is calculated respectively to obtain the clustering category vector corresponding to each image feature vector.

[0093] In one embodiment, the image feature vector and the corresponding cluster category vector are superimposed to obtain a fused feature vector of each search image, including: Subtract the corresponding cluster category vector from each image feature vector to obtain the fused feature vector of each search image.

[0094] In one embodiment, the image feature vectors are adaptively clustered to obtain multiple image clustering categories, including: In the process of adaptively clustering the image feature vector, the cluster center of the image feature vector is continuously adjusted; When the cluster centers meet the preset clustering condition, the image clustering category corresponding to each cluster center is determined.

[0095] In one embodiment, obtaining a first text vector of a search text and an image feature vector of each search image in an image database includes: Get all search images in the image database; Perform feature extraction on each search image respectively to obtain an image feature vector of each search image; When the search text is obtained, feature extraction is performed on the search text to obtain a first text vector of the search text.

[0096] The present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor in a computer device, the computer device can perform the steps of any embodiment of an image search method disclosed in the present invention. The computer-readable storage medium can be non-volatile or volatile.

[0097] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Obtaining a first text vector of the search text and an image feature vector of each search image in an image database; Adaptively cluster the image feature vectors to obtain the cluster category vector of each search image; Superimpose the image feature vector and the corresponding cluster category vector to obtain the fused feature vector of each search image; Calculate the similarity of the first feature of the search text vector and each fused feature vector respectively; According to the numerical value of the first feature similarity, a searched target image is determined from the search images.

[0098] In one embodiment, determining the searched target image from the search image according to the numerical value of the first feature similarity includes: Determining a preset number of candidate images from the search image according to the numerical value of the first feature similarity; Obtaining image information text of each candidate image, and extracting a second text vector of each image information text; Calculate the similarity between the second text vector and the image feature vector of each candidate image and the second feature of the first text vector; The candidate image corresponding to the second feature similarity with the largest value is determined as the target image.

[0099] In one embodiment, calculating the similarity between the second text vector and the image feature vector of each candidate image and the second feature of the first text vector includes: Calculating the text similarity between the second text vector and the first text vector, and calculating the image-text similarity between the image feature vector and the first text vector; Based on the text similarity and the image-text similarity, the second feature similarity is calculated.

[0100] In one embodiment, adaptive clustering is performed on the image feature vector to obtain a cluster category vector for each search image, including: Adaptively cluster the image feature vectors to obtain multiple image clustering categories; The vector average value of the image feature vector corresponding to each image clustering category is calculated respectively to obtain the clustering category vector corresponding to each image feature vector.

[0101] In one embodiment, the image feature vector and the corresponding cluster category vector are superimposed to obtain a fused feature vector of each search image, including: Subtract the corresponding cluster category vector from each image feature vector to obtain the fused feature vector of each search image.

[0102] In one embodiment, the image feature vectors are adaptively clustered to obtain multiple image clustering categories, including: In the process of adaptively clustering the image feature vector, the cluster center of the image feature vector is continuously adjusted; When the cluster centers meet the preset clustering conditions, the image clustering category corresponding to each cluster center is determined.

[0103] In one embodiment, obtaining a first text vector of a search text and an image feature vector of each search image in an image database includes: Get all search images in the image database; Perform feature extraction on each search image respectively to obtain an image feature vector of each search image; When the search text is obtained, feature extraction is performed on the search text to obtain a first text vector of the search text.

[0104] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0105] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0106] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An image search method, characterized in that: The method comprises: Obtaining a first text vector of the search text and an image feature vector of each search image in an image database; Adaptively clustering the image feature vectors to obtain a cluster category vector for each of the search images; Superimposing the image feature vector and the corresponding cluster category vector to obtain a fused feature vector of each search image; Calculating the first feature similarity of the search text vector and each of the fused feature vectors respectively; According to the numerical value of the first feature similarity, a searched target image is determined from the search images.

2. The image search method according to claim 1, characterized in that: Determining the searched target image from the search image according to the numerical value of the first feature similarity includes: Determining a preset number of candidate images from the search image according to the numerical value of the first feature similarity; Acquire the image information text of each candidate image, and extract the second text vector of each image information text; Calculating the second feature similarity between the second text vector and the image feature vector of each candidate image and the first text vector; The candidate image corresponding to the second feature similarity with the largest value is determined as the target image.

3. The image search method according to claim 2, characterized in that: The calculating the second text vector and the image feature vector of each candidate image and the second feature similarity of the first text vector comprises: Calculating the text similarity between the second text vector and the first text vector, and calculating the image-text similarity between the image feature vector and the first text vector; The second feature similarity is calculated based on the text similarity and the image-text similarity.

4. The image search method according to claim 1, wherein: The adaptively clustering the image feature vectors to obtain a clustering category vector for each search image includes: Adaptively clustering the image feature vector to obtain multiple image clustering categories; The vector average value is calculated for each of the image feature vectors corresponding to each of the image cluster categories to obtain a cluster category vector corresponding to each of the image feature vectors.

5. The image search method according to claim 4, characterized in that: The superimposing the image feature vector and the corresponding cluster category vector to obtain a fused feature vector of each search image includes: The corresponding cluster category vector is subtracted from each image feature vector to obtain a fused feature vector of each search image.

6. The image search method according to claim 4, characterized in that: The adaptive clustering of the image feature vectors to obtain multiple image clustering categories includes: In the process of adaptively clustering the image feature vectors, the clustering centers of the image feature vectors are continuously adjusted; When the cluster centers meet the preset clustering condition, the image clustering category corresponding to each cluster center is determined.

7. The image search method according to claim 1, wherein: The step of obtaining a first text vector of the search text and an image feature vector of each search image in the image database includes: Acquire all search images in the image database; Performing feature extraction on each of the search images respectively to obtain the image feature vector of each of the search images; When the search text is acquired, feature extraction is performed on the search text to obtain the first text vector of the search text.

8. An image search device, characterized in that: The device comprises: A vector acquisition module, used to acquire a first text vector of the search text and an image feature vector of each search image in an image database; An adaptive clustering module, used for adaptively clustering the image feature vectors to obtain a clustering category vector for each search image; A feature fusion module, used for superimposing the image feature vector and the corresponding cluster category vector to obtain a fused feature vector of each search image; A similarity calculation module, used for respectively calculating the first feature similarity between the search text vector and each of the fused feature vectors; The image search module is used to determine the searched target image from the search image according to the numerical value of the first feature similarity.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image search method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image search method according to any one of claims 1 to 7 is implemented.

Images