Database construction method, image retrieval method, and related device

Abstract

This invention provides a database construction method, an image retrieval method, and related apparatus. The database construction method includes: acquiring an image set, wherein each input image in the image set is labeled with its spatiotemporal domain information, associated spatiotemporal domain information corresponding to the input image, and additional spatiotemporal domain information of the input image; the associated spatiotemporal domain information is the spatiotemporal domain information corresponding to a target in the input image, and the additional spatiotemporal domain information is the spatiotemporal domain information where the target does not appear; determining the key spatiotemporal domain information based on the image set using an attention network model; and determining the database based on the key spatiotemporal domain information. This method determines the database used for image retrieval based on spatiotemporal domain information, which helps to address the situation where incorrect images within the archive lead to erroneous searches during subsequent retrieval processes, maintains the purity of the archive, and ensures accuracy and recall during image retrieval.

Description

Technical Field

[0001] This invention relates to the field of image retrieval technology, and in particular to a database construction method, an image retrieval method, and related apparatus. Background Technology

[0002] With the development of related technologies, face clustering has been widely applied. For example, in mobile phone photo albums, face clustering methods can group facial images of the same person into one group. However, traditional clustering networks, because they globally cluster all captured portrait images, often result in situations where one image contains multiple people, or one person contains multiple images. This leads to incorrect searches when using a target person's image, as mismatched images within the file can cause errors, compromising the accuracy and recall rate of image-based searches. Currently, no effective solution has been proposed to address these issues. Summary of the Invention

[0003] This invention provides a database construction method, an image retrieval method, and related devices. This method can solve the problem of incorrect images in archives leading to erroneous file searches, maintain the purity of the archives, and ensure the accuracy and recall rate of image searches.

[0004] To address the aforementioned technical problems, the first technical solution provided by this invention is: a method for constructing a database, comprising: acquiring an image set, wherein each input image in the image set is labeled with spatiotemporal domain information of the input image, associated spatiotemporal domain information corresponding to the input image, and additional spatiotemporal domain information of the input image; the associated spatiotemporal domain information is the spatiotemporal domain information corresponding to a target in the input image, and the additional spatiotemporal domain information is the spatiotemporal domain information where the target does not appear; determining the key spatiotemporal domain information based on the image set using an attention network model; and determining the database based on the key spatiotemporal domain information.

[0005] The step of determining the database based on the key spatiotemporal domain information includes: obtaining an image set corresponding to the key spatiotemporal domain information; clustering the image set to obtain multiple clusters; determining the similarity between the multiple clusters; establishing associations between clusters with similarity greater than a threshold; and thus obtaining the database.

[0006] The step of determining the similarity between multiple clusters, establishing associations between clusters with similarity greater than a threshold, and thus obtaining the database includes: determining the average centroid of each cluster; calculating the cosine similarity between each pair of clusters, establishing associations between clusters with cosine similarity greater than a threshold, and thus obtaining the database.

[0007] The step of determining the key spatiotemporal information based on the image set using an attention network model includes: obtaining first feature information corresponding to each input image, second feature information corresponding to the spatiotemporal information, third feature information corresponding to the target where each associated spatiotemporal information appears, and fourth feature information corresponding to the additional spatiotemporal information based on the image set; and determining the key spatiotemporal information based on the first feature information, the second feature information, the third feature information, and the fourth feature information.

[0008] The step of determining the key spatiotemporal information based on the first feature information, the second feature information, the third feature information, and the fourth feature information includes: calculating a first weight based on the first feature information and the second feature information; calculating a second weight based on the second feature information and the third feature information; calculating a third weight based on the second feature information and the fourth feature information; performing a weighted summation of the first weight, the second weight, and the third weight to obtain a weight result for each input image; and determining the key spatiotemporal information based on the weight result for each input image.

[0009] The step of determining the key spatiotemporal information based on the weight result of each input image includes: comparing the weight result of each input image with a weight threshold, associating the spatiotemporal information of input images whose weight result is greater than the weight threshold, and the associated spatiotemporal information is the key spatiotemporal information.

[0010] To address the aforementioned technical problems, the second technical solution provided by this invention is: a database construction apparatus, comprising: an acquisition module for acquiring an image set, wherein each input image in the image set is labeled with spatiotemporal domain information of the input image, associated spatiotemporal domain information corresponding to the input image, and additional spatiotemporal domain information of the input image; the associated spatiotemporal domain information is the spatiotemporal domain information corresponding to a target in the input image, and the additional spatiotemporal domain information is the spatiotemporal domain information where the target does not appear; a processing module for determining the key spatiotemporal domain information based on the image set using an attention network model; and a determination module for determining the database based on the key spatiotemporal domain information.

[0011] To address the aforementioned technical problems, the third technical solution provided by this invention is: providing an image retrieval method, comprising: acquiring feature information of an image to be retrieved; comparing the similarity of the feature information of the image to be retrieved with the centroid of each cluster in a database; the database being obtained by the database construction method described above; using the clusters whose similarity meets preset conditions as a candidate set; and determining a matching image for the image to be retrieved based on the candidate set.

[0012] The step of selecting the clusters whose similarity meets the preset conditions as a candidate set includes: selecting the clusters whose similarity meets the preset conditions, and the clusters associated with the clusters as the candidate set.

[0013] The step of comparing the similarity between the feature information of the image to be retrieved and the centroid of each cluster in the database includes: calculating the similarity between the feature information of the image to be retrieved and the average centroid of the cluster to obtain a first similarity; calculating the similarity between the feature information of the image to be retrieved and the recommended centroid of the cluster to obtain a second similarity; the step of selecting the clusters whose similarity meets the preset conditions as a candidate set includes: selecting the clusters whose first similarity and / or second similarity meet the preset conditions as a candidate set.

[0014] The step of determining the matching image of the image to be retrieved based on the candidate set includes: calculating the similarity between the feature information of the image to be retrieved and each image in the candidate set; and determining the matching image of the image to be retrieved based on the similarity.

[0015] To address the aforementioned technical problems, the fourth technical solution provided by this invention is: an image retrieval device, comprising: an acquisition module for acquiring feature information of an image to be retrieved; a comparison module for comparing the similarity of the feature information of the image to be retrieved with the centroid of each cluster in a database; the database being obtained by the database construction method described above; a candidate set determination module for selecting clusters whose similarity meets preset conditions as candidate sets; and a retrieval module for determining matching images of the image to be retrieved based on the candidate sets.

[0016] To solve the above-mentioned technical problems, the fifth technical solution provided by the present invention is: to provide an electronic device, including a processor and a memory coupled to each other, wherein the memory is used to store program instructions for implementing the method of any of the above-mentioned methods; and the processor is used to execute the program instructions stored in the memory.

[0017] To solve the above-mentioned technical problems, the sixth technical solution provided by the present invention is: to provide a computer-readable storage medium storing a program file, wherein the program file can be executed to implement any of the above methods.

[0018] The beneficial effects of this invention, distinct from existing technologies, lie in its method's use of an attention network model to determine key spatiotemporal information based on an image set. Then, a database is determined based on this key spatiotemporal information. Each input image in the image set is labeled with its spatiotemporal information, associated spatiotemporal information, and additional spatiotemporal information. This method determines the database used for image retrieval based on spatiotemporal information, thus mitigating the risk of incorrect image searches due to faulty files within the archive, maintaining the integrity of the archive, and ensuring high accuracy and recall during image searches. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the database construction method of the present invention;

[0021] Figure 2 This is a flowchart illustrating one embodiment of step S12;

[0022] Figure 3 This is a flowchart illustrating one embodiment of step S22;

[0023] Figure 4 This is a flowchart illustrating one embodiment of step S13;

[0024] Figure 5 This is a schematic diagram of the structure of an embodiment of the database construction apparatus of the present invention;

[0025] Figure 6 This is a flowchart illustrating an embodiment of the image retrieval method of the present invention;

[0026] Figure 7 This is a schematic diagram of the structure of an embodiment of the image retrieval device of the present invention;

[0027] Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention;

[0028] Figure 9This is a schematic diagram of the structure of the computer-readable storage medium of the present invention. Specific implementation methods

[0029] Existing technologies provide a method, device, and storage medium for image clustering. This method discloses an image clustering approach that integrates clustering results of different image features by fusing face, body, face-body association, body gait, and body gait association encoding. Suspected images are then weighted for secondary verification, resulting in accurate classification of suspected images, elimination of blurry areas, and improved recall. However, this method cannot effectively prevent misclassification of suspected images from different spatial domains within the same time domain. Furthermore, the weights are manually set, resulting in poor adaptability to different data distributions and insufficient algorithm universality.

[0030] Existing technologies also provide a method, apparatus, and application system for image search. This method extracts feature points from the search image, maps these feature points to classes in an image class index table, and selects the class containing the most feature points as the image's class. Searching is then performed within all images under that class, avoiding the need to traverse common points in each search and improving search efficiency. However, this method pre-builds an index. If the data distribution of the search image does not belong to the training data used in the pre-built index, the initially selected class may not contain the target image, resulting in low recall. Furthermore, this method maps the image's feature points to the index table and performs a brute-force search on images under the class with the most corresponding feature points. When several classes have the same number of feature points for the search image, this method leads to an excessive number of candidate target images to search in subsequent brute-force searches, causing time consumption to increase linearly. This method only uses the image's intrinsic information when extracting feature points, without utilizing its attribute information. Attribute information can guide the pre-index class division and improve the recall rate of subsequent searches.

[0031] Existing technology also provides an image search method. This method uses a similarity search category for the search image, performs feature encoding on the image using a hash encoding model corresponding to the similarity search category, thereby obtaining the target image of the search image and sending it to the user's visible terminal page. This patent requires the user to manually set the similarity search category and use the hash model within that category to extract the feature codes of the search images belonging to that category. This can lead to insufficient representativeness of the extracted feature codes if the data used to train the hash model for that category has weak similarity to the search image (possibly due to a discrepancy between the user's interest in the search image and the image's actual features). Directly comparing similarity within the user-specified category after extracting the feature codes can be memory and time-consuming when the number of images in that category is large. However, increasing the compression ratio of the hash model can lead to excessive image feature compression, resulting in feature loss and reduced recall.

[0032] This application provides a database construction method, an image retrieval method, and related apparatus. The method addresses the issue that traditional portrait clustering uses all captured images for archiving, which can lead to erroneous files during the search due to incorrect images within certain archives. Since the appearance of a target person has a fixed temporal and spatial range, this method pre-defines the spatiotemporal domain using the target person's image, then performs portrait clustering within the key spatiotemporal domain before searching. This improves the accuracy and recall of traditional clustering while maintaining the purity of the archives. The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0033] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] Please see Figure 1 The above is a flowchart illustrating the first embodiment of the database construction method of the present invention, specifically including:

[0035] Step S11: Obtain an image set, wherein each input image in the image set is labeled with the spatiotemporal domain information of the input image, the associated spatiotemporal domain information of the input image, and the additional spatiotemporal domain information of the input image.

[0036] Specifically, an image set is obtained, which includes several input images. Each input image is labeled with its spatiotemporal domain information, its associated spatiotemporal domain information, and its additional spatiotemporal domain information.

[0037] The spatiotemporal information of the input image includes the shooting location and shooting time. The associated spatiotemporal information is the spatiotemporal information corresponding to the target in the input image; that is, the associated spatiotemporal information is the shooting location and shooting time of the target in the input image. For example, if the target in the input image is A, then the associated spatiotemporal information is the spatiotemporal information of all captured images of target A. The additional spatiotemporal information is the spatiotemporal information where the target does not appear; for example, if the target in the input image is A, then the additional spatiotemporal information is the spatiotemporal information of captured images that do not contain target A.

[0038] Spatiotemporal information can be decomposed into a temporal domain and a spatial domain. The spatial domain refers to the correlation between checkpoints located at similar geographical locations or with specific characteristics. If a person is captured by a checkpoint, the probability of that person being captured by other checkpoints associated with that checkpoint increases. The temporal domain refers to the temporal patterns of a person's actions within the spatial domain. A person captured by a checkpoint at a particular time will have a higher probability of being captured by related checkpoints within that timeframe due to temporal correlations.

[0039] This application uses an image set to determine the key spatiotemporal and spatial domains.

[0040] Step S12: Use an attention network model to determine the key spatiotemporal information based on the image set.

[0041] Specifically, this application utilizes an attention network model to determine key spatiotemporal information based on an image set. Due to the correlation between different spatiotemporal information and the varying key spatiotemporal attention of the input image's spatiotemporal information, this application employs Deep Spatial-Temporal Neural Networks to process the image set and thus determine the key spatiotemporal information.

[0042] Please combine Figure 2 , Figure 2 A flowchart illustrating one embodiment of step S12 includes:

[0043] Step S21: Based on the image set, obtain the first feature information corresponding to each input image, the second feature information corresponding to the spatiotemporal domain information, the third feature information corresponding to the target where each associated spatiotemporal domain information appears, and the fourth feature information corresponding to the additional spatiotemporal domain information.

[0044] In one embodiment, the attention network model obtains first feature information X1 corresponding to each input image, second feature information X2 corresponding to the spatiotemporal information, third feature information X3 corresponding to each target in which the associated spatiotemporal information appears, and fourth feature information X4 corresponding to the additional spatiotemporal information based on the image set. Wherein, the first feature information X1 consists of n c-dimensional feature vectors; the second feature information X2 consists of t-dimensional feature vectors; the third feature information X3 consists of n l-dimensional feature vectors; and the fourth feature information X4 consists of n u-dimensional feature vectors.

[0045] Step S22: Determine the key spatiotemporal domain information based on the first feature information, the second feature information, the third feature information, and the fourth feature information.

[0046] Specifically, the key spatiotemporal domain information is determined based on the first feature information X1, the second feature information X2, the third feature information X3, and the fourth feature information X4.

[0047] Since the goal of this method is to find attention relationships / associations between spatiotemporal domains, it proposes to use the attention mechanism in neural networks to assign weights between different spatiotemporal domains.

[0048] For specific details, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating one embodiment of step S22, which specifically includes:

[0049] Step S31: Calculate a first weight based on the first feature information and the second feature information; calculate a second weight based on the second feature information and the third feature information; calculate a third weight based on the second feature information and the fourth feature information.

[0050] Specifically, a first weight is calculated based on the first feature information X1 and the second feature information X2; a second weight is calculated based on the second feature information X2 and the third feature information X3; and a third weight is calculated based on the second feature information X2 and the fourth feature information X4. The weight calculation formula is as follows:

[0051]

[0052] ∝ ci (m,n)=exp(h T ReLU(W tc [m,n]+b1)+b2);

[0053] Among them, W tc[m,n] represents the weights learned by the attention network from inputs m and n, b1 is the penalty coefficient, ReLU is the activation function that converts values ​​less than 0 to 0 and leaves values ​​greater than 0 unchanged, h T b1 represents the coefficients hidden for the attention network portion, b2 represents the penalty coefficient for the hidden layer calculation results, ∝ ci (m,n) represents the weight of m relative to n.

[0054] When calculating the first weight, m ​​is the second feature X2 and n is the first feature information X1; when calculating the second weight, m ​​is the second feature X2 and n is the third feature information X3; when calculating the third weight, m ​​is the second feature X2 and n is the fourth feature information X4.

[0055] Step S32: Perform a weighted summation of the first weight, the second weight, and the third weight to obtain the weight result for each input image.

[0056] Specifically, the first weight, the second weight, and the third weight are weighted and summed to obtain the weight result for each input image. This weight result represents the degree of correlation between the remaining spatiotemporal information and the spatiotemporal information of the input image.

[0057] In one embodiment, weight coefficients for a first weight, a second weight, and a third weight are further calculated. The first weight, the second weight, and the third weight are then weighted and summed based on the calculated weight coefficients to obtain the weight result for each input image.

[0058] It should be noted that the attention network model in this application differs from existing technologies in that the weight coefficients of the first, second, and third weights are not normalized to the range of 0-1 using softmax. This method takes into account the spatiotemporal information of the input image, thus allowing the weights of each spatiotemporal domain to be adaptively adjusted for different domains. Secondly, this weight calculation method optimizes a problem in the original attention network model's weight calculation method: when a certain spatiotemporal domain has little correlation with other spatiotemporal domains, the weights of some spatiotemporal domains will not become excessively large due to normalization. Finally, due to the attention / correlation relationships between spatiotemporal domains, when the spatiotemporal information of the input image differs, the weights of the spatiotemporal domains associated with that domain should be increased accordingly. Therefore, for different spatiotemporal information of the input image, the weights of other spatiotemporal domains will continuously change, and since the weights are not normalized, the various problems caused by normalization are avoided.

[0059] Step S33: Determine the key spatiotemporal information based on the weight result of each input image.

[0060] Specifically, for each input image in the image set, a corresponding weight result is calculated. The weight result of each input image is compared with a weight threshold. The spatiotemporal information of input images whose weight results are greater than the weight threshold is correlated. The correlated spatiotemporal information is the key spatiotemporal information. For example, if the input images a, b, c, and d have weight results greater than the weight threshold, then the spatiotemporal information of input images a, b, c, and d is correlated to obtain the key spatiotemporal information.

[0061] Step S13: Determine the database based on the key spatiotemporal domain information.

[0062] Specifically, the database is determined based on key spatiotemporal domain information. Please refer to... Figure 4 , Figure 4 This is a flowchart illustrating one embodiment of step S13, which specifically includes:

[0063] Step S41: Obtain the image set corresponding to the key spatiotemporal domain information.

[0064] Obtain the image set corresponding to key spatiotemporal information. Specifically, given the key spatiotemporal information, we can extract the images corresponding to the capture location and capture time.

[0065] Step S42: Cluster the image set to obtain multiple clusters.

[0066] The image set is clustered to obtain multiple clusters. The determination of the key time domains mentioned above ensures the purity of the images to be clustered to a certain extent. Further clustering of the image set yields the images to be clustered. After clustering, multiple clusters are obtained. This clustering process ensures the purity of the files after clustering, reducing the probability of one person having multiple files or one file having multiple users.

[0067] Step S43: Determine the similarity between multiple clusters, establish associations between clusters with similarity greater than a threshold, and thus obtain the database.

[0068] Specifically, the average centroid of each cluster is determined. After clustering into multiple clusters, the average centroid of each cluster is determined. Cosine similarity is calculated between each pair of clusters, and clusters with cosine similarity greater than a threshold are associated, thus obtaining the database. Specifically, based on the average centroid, cosine similarity is calculated between each pair of clusters, and clusters with cosine similarity greater than a threshold are associated, thus obtaining the database. This establishes connectivity relationships between clusters within key spatiotemporal domains, providing guidance for subsequent image retrieval.

[0069] The database construction method of this application utilizes image attributes, especially spatiotemporal information, to cluster images after dividing them into key spatiotemporal domains. This reduces the probability of incorrect images being clustered and significantly improves the recall rate during subsequent searches. When dividing key spatiotemporal domains, an interactive self-attention DSTN deep model based on an adaptive attention mechanism is used. Therefore, the association between different spatiotemporal domains can be adaptively adjusted based on the input target spatiotemporal domain. This method is not limited to the time and spatial domains (nearby locations and similar times) physically associated with the target spatiotemporal domain, which can improve the recall rate of subsequent searches. By dividing key spatiotemporal domains (target image spatiotemporal domains associated with the spatiotemporal domain of the image), the time and resource consumption of subsequent clustering steps can be greatly reduced. Therefore, more accurate methods can be used for searching, ensuring greater accuracy.

[0070] In real-world data, utilizing the inherent attributes of the data itself is a prerequisite for obtaining good search results. Because the same person has a very high probability of appearing in the associated / key spatiotemporal domains of the searched images, pre-defining key spatiotemporal domains using the spatiotemporal domain information of the images can reduce the likelihood of incorrect images being included in cluster results / files, thus improving search recall. When extracting key spatiotemporal domains, using an adaptive attention mechanism model can divide key spatiotemporal domains beyond their physically associated spatiotemporal domains (captures at nearby checkpoints, at the same checkpoint at similar times, or at related checkpoints). Considering the different weights of attention mechanisms for other spatiotemporal domains in different target spatiotemporal domains, re-clustering spatiotemporal domains with high attention weights as key spatiotemporal domains can cover all images of that target person, thus improving search recall. Meanwhile, clustering within key spatiotemporal domains ensures that the average and recommended centroids of each cluster better represent the image features within each cluster / file in the spatiotemporal domain of the searched image (due to the similarity of external lighting conditions in similar spatiotemporal domains, some redundant information can be removed), thus improving the accuracy of the clustering results. Secondly, establishing connectivity between clusters prevents potential clustering errors caused by some data falling on the boundaries between two clusters, eliminates the possibility of unrecalled images due to target images being in ambiguous areas, and improves the recall rate during image search. Finally, in the file clustering step, limiting target images to key spatiotemporal domains significantly reduces the time and space resources required when using the same image search strategy compared to using all images.

[0071] Please see Figure 5 This is a schematic diagram of an embodiment of the database construction apparatus of the present invention. It includes: an acquisition module 51, a processing module 52, and a determination module 53.

[0072] The acquisition module 51 is used to acquire an image set, wherein each input image in the image set is labeled with the spatiotemporal domain information of the input image, the associated spatiotemporal domain information corresponding to the input image, and the additional spatiotemporal domain information of the input image; the associated spatiotemporal domain information is the spatiotemporal domain information corresponding to the target in the input image, and the additional spatiotemporal domain information is the spatiotemporal domain information where the target does not appear.

[0073] The processing module 52 is used to determine the key spatiotemporal information based on the image set using an attention network model.

[0074] In one embodiment, the processing module 52 is further configured to obtain, based on the image set, first feature information corresponding to each input image, second feature information corresponding to the spatiotemporal domain information, third feature information corresponding to the target where each associated spatiotemporal domain information appears, and fourth feature information corresponding to the additional spatiotemporal domain information; and determine the key spatiotemporal domain information based on the first feature information, the second feature information, the third feature information, and the fourth feature information.

[0075] In one embodiment, the processing module 52 is further configured to calculate a first weight based on the first feature information and the second feature information; calculate a second weight based on the second feature information and the third feature information; calculate a third weight based on the second feature information and the fourth feature information; perform a weighted summation of the first weight, the second weight, and the third weight to obtain a weight result for each input image; and determine the key spatiotemporal domain information based on the weight result for each input image.

[0076] In one embodiment, the processing module 52 is further configured to compare the weight result of each input image with a weight threshold, and associate the spatiotemporal information of the input images whose weight result is greater than the weight threshold, wherein the associated spatiotemporal information is the key spatiotemporal information.

[0077] The determination module 53 is used to determine the database based on the key spatiotemporal domain information.

[0078] In one embodiment, the determining module 53 is further configured to obtain an image set corresponding to the key spatiotemporal domain information; cluster the image set to obtain multiple clusters; determine the similarity between the multiple clusters; establish an association between clusters with similarity greater than a threshold; and thereby obtain the database.

[0079] In one embodiment, the determining module 53 is further configured to determine the average centroid of each of the said clusters; calculate the cosine similarity between each pair of clusters; establish associations between clusters with cosine similarity greater than a threshold; and thus obtain the database.

[0080] Please see Figure 6 The above is a flowchart illustrating an embodiment of the image retrieval method of the present invention, specifically including:

[0081] Step S61: Obtain the feature information of the image to be retrieved.

[0082] Specifically, feature extraction is performed on the image to be retrieved to obtain its feature information.

[0083] Step S62: Compare the similarity between the feature information of the image to be retrieved and the centroid of each cluster in the database.

[0084] Specifically, the similarity between the feature information of the image to be retrieved and the centroid of each cluster in the database is calculated. It should be noted that the database is obtained through the above... Figures 1-4 The database was obtained using the method shown.

[0085] Step S63: Select the clusters whose similarity meets the preset conditions as a candidate set.

[0086] In one specific embodiment, the similarity between the feature information of the image to be retrieved and the average centroid of each cluster in the database is calculated to obtain a first similarity. If the first similarity meets the preset conditions, the corresponding cluster is used as a candidate set.

[0087] In another embodiment, the similarity between the feature information of the image to be retrieved and the recommended centroid of the cluster is calculated to obtain a second similarity. If the second similarity meets a preset condition, the corresponding cluster is selected as a candidate set. It should be noted that the recommended centroid is the x images with the highest quality scores in the corresponding cluster.

[0088] In another embodiment, the similarity between the feature information of the image to be retrieved and the average centroid of the cluster is calculated to obtain a first similarity; the similarity between the feature information of the image to be retrieved and the recommended centroid of the cluster is calculated to obtain a second similarity; and the clusters whose first similarity and second similarity meet the preset conditions are selected as a candidate set.

[0089] Understandably, similarity calculation methods are not limited to cosine similarity.

[0090] In another embodiment of this application, the clusters whose similarity meets preset conditions, as well as the clusters associated with the clusters, are used as the candidate set. Specifically, during database construction, associations between clusters are established based on the similarity between clusters. Then, during image retrieval, other clusters associated with the clusters can be filtered out based on these associations and also used as the candidate set, thereby improving the recall rate.

[0091] Step S64: Determine the matching image of the image to be retrieved based on the candidate set.

[0092] Calculate the similarity between the feature information of the image to be retrieved and each image in the candidate set; determine the matching image of the image to be retrieved based on the similarity.

[0093] Specifically, the search image and the target feature encoding in the candidate set are compared by cosine similarity. The target files within a set number / with a confidence level higher than a certain threshold are returned on the user display terminal in descending order of similarity. This yields the matching image of the image to be retrieved.

[0094] Please see Figure 7 This is a schematic diagram of the structure of an embodiment of the image retrieval device of the present invention, specifically including: an acquisition module 71, a comparison module 72, a candidate set determination module 73, and a retrieval module 74.

[0095] The acquisition module 71 is used to acquire the feature information of the image to be retrieved.

[0096] The comparison module 72 is used to compare the feature information of the image to be retrieved with the centroid of each cluster in the database.

[0097] The database is described above. Figures 1 to 4 The database is obtained from the database construction method described in any embodiment.

[0098] In one embodiment, the comparison module 72 is further configured to calculate the similarity between the feature information of the image to be retrieved and the average centroid of the cluster to obtain a first similarity; and calculate the similarity between the feature information of the image to be retrieved and the recommended centroid of the cluster to obtain a second similarity.

[0099] The candidate set determination module 73 is used to select the clusters whose similarity meets the preset conditions as candidate sets.

[0100] In one embodiment, the candidate set determination module 73 is used to select the clusters whose first similarity and / or second similarity meet preset conditions as candidate sets.

[0101] In one embodiment, the candidate set determination module 73 uses the clusters whose similarity meets the preset conditions, as well as the clusters associated with the clusters, as the candidate set.

[0102] The retrieval module 74 is used to determine the matching image of the image to be retrieved based on the candidate set.

[0103] In one embodiment, the retrieval module 74 is used to calculate the similarity between the feature information of the image to be retrieved and each image in the candidate set; and to determine the matching image of the image to be retrieved based on the similarity.

[0104] Please see Figure 8 This is a schematic diagram of an embodiment of the electronic device of the present invention. The electronic device includes a memory 82 and a processor 81 connected to each other.

[0105] The memory 82 is used to store program instructions for implementing any of the above methods.

[0106] Processor 81 is used to execute program instructions stored in memory 82.

[0107] The processor 81 can also be referred to as a CPU (Central Processing Unit). The processor 81 may be an integrated circuit chip with signal processing capabilities. The processor 81 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0108] The memory module 82 can be a RAM stick, TF card, etc., and can store all information in the electronic device, including raw input data, computer programs, intermediate running results, and final running results. It stores and retrieves information according to the location specified by the controller. With memory, the electronic device has a memory function and can ensure normal operation. According to their purpose, the memory of an electronic device can be divided into main memory (RAM) and auxiliary memory (external storage), or it can be classified as external memory and internal memory. External storage is usually magnetic media or optical discs, which can store information for a long time. RAM refers to the storage components on the motherboard, used to store currently executing data and programs, but it is only used for temporary storage of programs and data; the data will be lost when the power is turned off.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented by other methods. For example, the apparatus implementation methods described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this implementation method according to actual needs.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a system server, or a network device, etc.) or a processor to execute all or part of the steps of the various embodiments of this application.

[0113] Please see Figure 9 This is a schematic diagram of the structure of the computer-readable storage medium of the present invention. The storage medium of this application stores a program file 91 capable of implementing all the above methods. This program file 91 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of each embodiment of the present application. The aforementioned storage devices include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0114] The above are merely implementation methods of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for constructing a database, characterized in that, include: Obtain an image set, wherein each input image in the image set is labeled with the spatiotemporal domain information of the input image, the associated spatiotemporal domain information of the input image, and the additional spatiotemporal domain information of the input image; The associated spatiotemporal domain information is the spatiotemporal domain information corresponding to the target in the input image, and the additional spatiotemporal domain information is the spatiotemporal domain information where the target does not appear. The weight result of each input image is calculated using an attention network model. Based on the weight result of each input image, the spatiotemporal information of the input images is correlated to determine the key spatiotemporal information. The weight result represents the degree of correlation between the remaining spatiotemporal information and the spatiotemporal information of the input images. The database is determined based on the key spatiotemporal domain information.

2. The construction method according to claim 1, characterized in that, The step of determining the database based on the key spatiotemporal domain information includes: Obtain the image set corresponding to the key spatiotemporal information; The image set is clustered to obtain multiple clusters; The similarity between multiple clusters is determined, and clusters with similarity greater than a threshold are associated to obtain the database.

3. The construction method according to claim 2, characterized in that, The step of determining the similarity between multiple clusters, establishing associations between clusters with similarity greater than a threshold, and thereby obtaining the database includes: Determine the average centroid of each of the aforementioned clusters; The cosine similarity between each pair of clusters is calculated, and clusters with a cosine similarity greater than a threshold are associated to obtain the database.

4. The construction method according to any one of claims 1 to 3, characterized in that, The step of calculating the weights for each input image using an attention network model includes: Based on the image set, we obtain the first feature information corresponding to each input image, the second feature information corresponding to the spatiotemporal domain information, the third feature information corresponding to the target where each associated spatiotemporal domain information appears, and the fourth feature information corresponding to the additional spatiotemporal domain information. A first weight is calculated based on the first feature information and the second feature information; a second weight is calculated based on the second feature information and the third feature information; a third weight is calculated based on the second feature information and the fourth feature information. The first weight, the second weight, and the third weight are weighted and summed to obtain the weight result for each input image.

5. The construction method according to claim 4, characterized in that, The step of associating the spatiotemporal information of the input images based on the weight results of each input image to determine the key spatiotemporal information includes: The weight result of each input image is compared with a weight threshold. The spatiotemporal information of the input images whose weight result is greater than the weight threshold is associated. The associated spatiotemporal information is the key spatiotemporal information.

6. A database construction apparatus, characterized in that, include: The acquisition module is used to acquire an image set, wherein each input image in the image set is labeled with the spatiotemporal domain information of the input image, the associated spatiotemporal domain information of the input image, and the additional spatiotemporal domain information of the input image; The associated spatiotemporal domain information is the spatiotemporal domain information corresponding to the target in the input image, and the additional spatiotemporal domain information is the spatiotemporal domain information where the target does not appear. The processing module is used to calculate the weight result of each input image using an attention network model, and to associate the spatiotemporal information of the input images based on the weight result of each input image, thereby determining the key spatiotemporal information. The weight result represents the degree of association between the remaining spatiotemporal information and the spatiotemporal information of the input image. The determination module is used to determine the database based on the key spatiotemporal domain information.

7. An image retrieval method, characterized in that, include: Obtain feature information of the image to be retrieved; The feature information of the image to be retrieved is compared with the centroid of each cluster in the database for similarity. The database is obtained by the database construction method according to any one of claims 1 to 5 above; The clusters whose similarity meets the preset conditions are used as the candidate set; The matching image for the image to be retrieved is determined based on the candidate set.

8. The image retrieval method according to claim 7, characterized in that, The step of selecting the clusters whose similarity meets the preset conditions as a candidate set includes: The clusters whose similarity meets the preset conditions, and the clusters associated with the clusters, are used as the candidate set.

9. The image retrieval method according to claim 7, characterized in that, The step of comparing the feature information of the image to be retrieved with the centroid of each cluster in the database includes: Calculate the similarity between the feature information of the image to be retrieved and the average centroid of the cluster to obtain the first similarity; Calculate the similarity between the feature information of the image to be retrieved and the recommended centroid of the cluster to obtain the second similarity; The step of selecting the clusters whose similarity meets the preset conditions as a candidate set includes: The clusters that meet the preset conditions for first similarity and / or second similarity are selected as the candidate set.

10. The image retrieval method according to claim 7, characterized in that, The step of determining the matching image of the image to be retrieved based on the candidate set includes: Calculate the similarity between the feature information of the image to be retrieved and each image in the candidate set; The matching image of the image to be retrieved is determined based on similarity.

11. An image retrieval device, characterized in that, include: The acquisition module is used to acquire feature information of the image to be retrieved; A comparison module is used to compare the feature information of the image to be retrieved with the centroid of each cluster in the database; the database is obtained by the database construction method according to any one of claims 1 to 5. The candidate set determination module is used to select the clusters whose similarity meets the preset conditions as the candidate set; The retrieval module determines the matching image of the image to be retrieved based on the candidate set.

12. An electronic device, characterized in that, This includes interconnected processors and memory, among which, The memory is used to store program instructions for implementing the method as described in any one of claims 1-5 and 7-10; The processor is used to execute the program instructions stored in the memory.

13. A computer-readable storage medium, characterized in that, The system contains a program file that can be executed to implement the method as described in any one of claims 1-5 and 7-10.

Images