Image retrieval method and model training method, device, equipment, medium and product

Abstract

The present disclosure provides an image retrieval method, a model training method, an apparatus, a device, a medium and a product, relates to the field of image processing, and in particular to the field of deep learning. The specific implementation scheme is: obtaining a to-be-retrieved image; extracting visual features of the to-be-retrieved image based on a visual retrieval model obtained through joint training of multiple training tasks; extracting semantic features of the to-be-retrieved image based on a semantic retrieval model obtained through joint training of multiple loss functions; performing feature splicing on the visual features and the semantic features to obtain target features of the to-be-retrieved image; and searching for a target image that meets an image similarity condition with the to-be-retrieved image from an image library based on the target features. The technical scheme of the present disclosure improves the image retrieval accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning in image processing, and more particularly to an image retrieval method, model training method, apparatus, device, medium, and product. Background Technology

[0002] Image retrieval technology aims to retrieve images from image databases that meet search criteria. Many related technologies directly search for images similar to the image being searched. Due to the vast size of image databases, to improve retrieval efficiency, images in the database can be initially divided into several categories, such as animals, plants, products, materials, text, and titles. When performing category-based retrieval on the image to be searched, image retrieval service processes need to be executed separately for each category, including training, using, and setting up the retrieval model for each category. Based on the retrieval results for each image category, a merging strategy is set to obtain the final search results. However, the search results obtained using this method are not accurate enough. Summary of the Invention

[0003] This disclosure provides an image retrieval method, model training method, apparatus, device, medium, and product for the field of image retrieval.

[0004] According to a first aspect of this disclosure, an image retrieval method is provided, comprising:

[0005] Obtain the image to be retrieved;

[0006] The visual retrieval model, obtained by joint training of multiple training tasks, extracts the visual features of the image to be retrieved.

[0007] The semantic retrieval model, obtained by joint training of multiple loss functions, extracts the semantic features of the image to be retrieved.

[0008] The visual features and the semantic features are concatenated to obtain the target features of the image to be retrieved;

[0009] Based on the target features, a target image that meets the image similarity condition to the image to be retrieved is searched from the image library.

[0010] According to a second aspect of this disclosure, a method for training a visual retrieval model is provided, comprising:

[0011] Based on the visual similarity clustering algorithm, multiple candidate images in the image library are visually clustered to obtain at least one visual category.

[0012] Generate a visual retrieval model to be trained, which includes a first encoder and multiple training tasks;

[0013] Determine at least one first training data; the first training data includes a first training image labeled with a visual ground truth and a second training image;

[0014] At least one of the first training data is input into the visual retrieval model to be trained, and the first loss value generated by multiple training tasks on the visual retrieval model as a whole satisfies the first error condition is taken as the training objective, and the visual model parameters of the visual retrieval model are obtained through training.

[0015] The first encoder of the visual retrieval model corresponding to the visual model parameters is used to extract the visual features of the image to be retrieved.

[0016] According to a third aspect of this disclosure, a method for training a semantic retrieval model is provided, comprising:

[0017] Based on the semantic category mining algorithm, semantic category mining is performed on multiple candidate images in the image library to obtain at least one semantic category;

[0018] Generate a semantic retrieval model to be trained, which includes a second encoder and a target training task; the target training task corresponds to multiple loss functions;

[0019] Determine at least one second training data; the second training data includes a third training image labeled with semantic category ground truth values;

[0020] At least one of the second training data is input into the semantic retrieval model to be trained, and the second loss value obtained by performing overall loss calculation on the semantic retrieval model based on multiple loss functions satisfies the second error condition as the training objective, thereby training to obtain the semantic model parameters of the semantic retrieval model;

[0021] The second encoder in the semantic retrieval model corresponding to the semantic model parameters is used to extract the semantic features of the image to be retrieved.

[0022] According to a fourth aspect of this disclosure, an image retrieval apparatus is provided, comprising:

[0023] An image acquisition unit is used to acquire the image to be retrieved.

[0024] The first extraction unit is used to extract the visual features of the image to be retrieved based on a visual retrieval model obtained by joint training of multiple training tasks.

[0025] The second extraction unit is used to extract the semantic features of the image to be retrieved based on the semantic retrieval model obtained by joint training of multiple loss functions.

[0026] The first splicing unit is used to splice the visual features and the semantic features to obtain the target features of the image to be retrieved.

[0027] The target search unit is used to search for target images in the image library that meet the image similarity conditions to the image to be searched, based on the target features.

[0028] According to a fifth aspect of this disclosure, a training apparatus for a visual retrieval model is provided, comprising:

[0029] The category clustering unit is used to perform visual clustering processing on multiple candidate images in the image library based on the visual similarity clustering algorithm to obtain at least one visual category;

[0030] The first generation unit is used to generate a visual retrieval model to be trained, which includes a first encoder and multiple training tasks.

[0031] A first determining unit is configured to determine at least one first training data; the first training data includes a first training image labeled with a visual ground truth and a second training image.

[0032] The first training unit is used to input at least one of the first training data into the visual retrieval model to be trained, with the training objective being that the first loss value generated by multiple training tasks on the overall visual retrieval model satisfies the first error condition, and to train and obtain the visual model parameters of the visual retrieval model.

[0033] The first encoder of the visual retrieval model corresponding to the visual model parameters is used to extract the visual features of the image to be retrieved, and the visual features are used to find the target image from the image library.

[0034] According to a sixth aspect of this disclosure, a training apparatus for a semantic retrieval model is provided, comprising:

[0035] The category mining unit is used to perform semantic category mining on multiple candidate images in the image library based on the semantic category mining algorithm to obtain at least one semantic category;

[0036] The second generation unit is used to generate a semantic retrieval model to be trained, which includes a second encoder and a target training task; the target training task corresponds to multiple loss functions.

[0037] The second determining unit is configured to determine at least one second training data; the second training data includes a third training image labeled with semantic category ground truth values;

[0038] The second training unit is used to input at least one of the second training data into the semantic retrieval model to be trained, with the second loss value obtained by performing overall loss calculation on the semantic retrieval model based on multiple loss functions satisfying the second error condition as the training objective, and to train and obtain the semantic model parameters of the semantic retrieval model.

[0039] The second encoder in the semantic retrieval model corresponding to the semantic model parameters is used to extract the semantic features of the image to be retrieved, and the semantic features are used to find the target image from the image library.

[0040] According to a seventh aspect of this disclosure, an electronic device is provided, comprising:

[0041] At least one processor; and

[0042] A memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of the first, second, or third aspects.

[0044] According to an eighth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method of any one of the first, second, or third aspects.

[0045] According to a ninth aspect of this disclosure, a computer program product is provided, the computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, the at least one processor executing the computer program causing the electronic device to perform the method of any one of the first, second, or third aspects.

[0046] The technology disclosed herein solves the problem of low accuracy in image retrieval results. By using target features, target images that meet the image similarity criteria to the image to be retrieved can be found in the image database. Semantic and visual features integrate both visual and semantic features of the image to be retrieved, resulting in more comprehensive features and thus enabling accurate identification of the image to be retrieved.

[0047] It should be understood that the descriptions in this section are not intended to label key or essential features of the embodiments of this disclosure, nor are they intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0048] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0049] Figure 1 This is an application diagram of an image retrieval method provided according to an embodiment of the present disclosure;

[0050] Figure 2 This is a flowchart of an image retrieval method provided according to an embodiment of the present disclosure;

[0051] Figure 3 This is another flowchart of an image retrieval method provided according to an embodiment of the present disclosure.

[0052] Figure 4 This is a flowchart of a training method for a visual retrieval model provided according to an embodiment of the present disclosure;

[0053] Figure 5 This is a flowchart of a training method for a semantic retrieval model provided according to an embodiment of the present disclosure;

[0054] Figure 6 This is a schematic diagram of an image retrieval device provided according to this public embodiment;

[0055] Figure 7 This is a schematic diagram of a training device for a visual retrieval model provided according to an embodiment of the present disclosure;

[0056] Figure 8 This is a schematic diagram of a training device for a semantic retrieval model provided according to an embodiment of the present disclosure;

[0057] Figure 9 This is a block diagram of an electronic device used to implement the image retrieval, visual retrieval model training, or semantic retrieval model training methods of the embodiments of this disclosure. Detailed Implementation

[0058] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0059] The technical solution disclosed herein can be applied to image retrieval scenarios. It extracts the visual features of the image to be retrieved through a visual retrieval model and the semantic features of the image to be retrieved through a semantic retrieval model, thereby achieving dual feature extraction of the image. This allows for the direct use of highly accurate features to achieve rapid image retrieval, improving the accuracy and efficiency of image retrieval.

[0060] In image retrieval technologies, category retrieval techniques are commonly used. This involves initially classifying images in a database to obtain a limited number of simple image categories, such as animals, plants, products, materials, text, and titles. In practical applications, a retrieval service needs to be established for each image category. This might involve training feature extraction and classification models for each category. Furthermore, based on the retrieval services established for each image category, it may be necessary to aggregate the retrieval results from multiple image categories and set merging strategies to complete the image retrieval process. However, this existing image retrieval method suffers from high retrieval service complexity, low retrieval costs, and low efficiency.

[0061] To address the aforementioned technical issues, this disclosure considers using feature extraction from two perspectives: visual and semantic features, enabling accurate extraction of image features. To extract accurate visual and semantic features, multiple training tasks can be employed to train a visual retrieval model accurately on the visual similarity characteristics of images, and a semantic retrieval model accurately extracting the semantic features of images. By accurately extracting image features, image retrieval can be directly performed using these features, thereby improving image retrieval accuracy.

[0062] In this embodiment, after obtaining the image to be retrieved, a visual retrieval model trained jointly by multiple training tasks can be used to extract the visual features of the image to be retrieved, and a semantic retrieval model trained jointly by multiple loss functions can be used to extract the semantic features of the image to be retrieved. The visual and semantic features are then concatenated to obtain the target features. These target features can be used to find target images in the image library that meet the image similarity criteria to the image to be retrieved. The semantic and visual features combine the visual and semantic aspects of the image to be retrieved, resulting in more comprehensive features and enabling accurate identification of the image to be retrieved.

[0063] This disclosure provides an image retrieval method, model training method, apparatus, device, medium, and product, which are applied to the field of deep learning in image processing to improve retrieval accuracy and efficiency.

[0064] For ease of understanding, the technical solution of this disclosure will be described in detail below with reference to the accompanying drawings.

[0065] like Figure 1 The diagram illustrates an application of an image retrieval method provided in this embodiment. In image retrieval applications, image retrieval can be a retrieval service provided externally. (Reference) Figure 1User device 1 can output a search page 1 for the user to search for an image. The user can input an image to be searched on the search page 1. When the search page 1 detects the image input by the user, it can send the image to an electronic device 2 configured with the image retrieval method provided in this disclosure. The electronic device 2 may include, for example, a computer, a laptop, a regular server, a cloud server, etc. Figure 1 The electronic device 2 shown is a cloud server. The cloud server 2 can be configured with a trained visual retrieval model 21 and a semantic retrieval model 22. The visual retrieval model 21 can extract the visual features of the image to be retrieved. The semantic retrieval model 22 can extract the semantic features of the image to be retrieved. After concatenating the visual and semantic features of the image to be retrieved to obtain the target features, the target image can be searched from the image library 23 using the target features. Then, the cloud server 2 can send the target image to the user device 1 to achieve fast and accurate acquisition of the target image.

[0066] like Figure 2 The diagram shown is a flowchart of an image retrieval method provided in this disclosure. This method can be configured as an image retrieval device, which can be installed in an electronic device. The image retrieval method may include the following steps:

[0067] 201: Retrieve the image to be searched.

[0068] The image to be retrieved can be a digital image, and can be any image format, such as Joint Photographic Experts Group (JPEG) format, Bitmap (BMP) format, Tag Image File Format (TIFF) format, etc. In this embodiment of the disclosure, no specific format of the image is limited.

[0069] 202: Based on a visual retrieval model obtained through joint training of multiple training tasks, visual features of the image to be retrieved are extracted.

[0070] Optionally, the steps for obtaining the visual retrieval model include: obtaining the visual parameter file of the visual retrieval model, reading the visual model parameters from the visual parameter file, loading the visual retrieval model based on the visual model parameters, and completing the acquisition of the visual retrieval model. The visual parameter file can be stored at the end of the visual retrieval model training. The training steps for the semantic retrieval model can be found in [reference needed]. Figure 4 The example shown.

[0071] 203: A semantic retrieval model obtained by joint training of multiple loss functions is used to extract semantic features of the image to be retrieved.

[0072] Optionally, the steps for obtaining the semantic retrieval model include: obtaining the semantic parameter file of the semantic retrieval model, reading the semantic model parameters from the semantic parameter file, loading the semantic retrieval model based on the semantic model parameters, and completing the acquisition of the semantic retrieval model. The semantic parameter file can be stored at the end of the semantic retrieval model training. The training steps for the semantic retrieval model can be found in [reference needed]. Figure 5 The example shown.

[0073] Jointly training a visual retrieval model using multiple training tasks refers to training the visual retrieval model simultaneously using multiple training tasks. Training tasks can refer to at least two of the following: classification, recognition, detection, and similarity measurement tasks performed on visual features.

[0074] Jointly training a semantic retrieval model using multiple loss functions refers to using multiple loss functions to comprehensively calculate the loss for the target training task of a visual retrieval model. These multiple loss functions can refer to at least two of the following loss functions: distance loss function, center loss function, cross-entropy loss function, etc.

[0075] 204: Visual features and semantic features are concatenated to obtain the target features of the image to be retrieved.

[0076] 205: Based on target features, search the image library for target images that meet the image similarity criteria to the image to be retrieved.

[0077] Optionally, candidate image features corresponding to multiple candidate images in the image library can be determined. Feature similarity calculations are then performed between the image to be retrieved and the features of the multiple candidate images to obtain the image similarity of the image to be retrieved in each of the multiple candidate images. The image similarities of the multiple candidate images are then sorted in descending order, and N candidate images are selected as the target image from the sorted list. Here, N is a positive integer greater than or equal to 1.

[0078] In this embodiment, after obtaining the image to be retrieved, multiple training tasks can be used in conjunction with a visual retrieval model to extract visual features of the image to be retrieved, and multiple training tasks can be used in conjunction with a semantic retrieval model to extract semantic features of the image to be retrieved. The visual and semantic features are then concatenated to obtain target features. These target features can be used to find target images in the image library that meet the image similarity criteria to the image to be retrieved. The semantic and visual features integrate both visual and semantic aspects of the image to be retrieved, resulting in more comprehensive features and enabling accurate identification of the image to be retrieved.

[0079] To obtain the image to be retrieved in real time, one possible design includes:

[0080] Receive image retrieval requests sent by user equipment;

[0081] In response to an image retrieval request, retrieve the image to be retrieved.

[0082] Optionally, the image retrieval service can be provided externally, such as providing a webpage link to an image retrieval page to the user. The user device can respond to the webpage link entered by the user in the browser, obtain the image retrieval page sent by the electronic device, and display the image retrieval page to the user. The user device can also detect the selection request triggered by the user's image input control on the image retrieval page, display an image path selection control to the user, and detect the image path provided by the user based on the image path selection control. Images are read according to the image path, and an image retrieval request is generated based on the images read from the image path. The image contained in the image retrieval request is the image to be retrieved. The user device can also send the image retrieval request to an electronic device. In practical applications, the user device may include terminal devices such as personal computers, mobile phones, and laptops; this disclosure does not impose excessive limitations on the specific type of user device.

[0083] In this embodiment of the disclosure, upon receiving an image retrieval request sent by a user equipment, the system can respond to the request and obtain the image to be retrieved. By interacting with the user, the image to be retrieved can be obtained in real time, providing external image retrieval services and improving service timeliness.

[0084] To extract accurate visual features, as an example, extracting visual features from the image to be retrieved based on a trained visual retrieval model may include:

[0085] Obtain the first encoder of the visual retrieval model obtained by joint training of multiple training tasks.

[0086] The image to be retrieved is input into the first encoder, and the visual features of the image to be retrieved are calculated using the first encoder.

[0087] A visual retrieval model may include a first encoder and multiple training tasks. These multiple training tasks can be composed of multiple tasks.

[0088] In this embodiment, the first encoder of the visual retrieval model, which employs multiple training tasks, performs accurate feature extraction on the image to be retrieved, thereby improving the efficiency and accuracy of feature extraction.

[0089] To extract accurate semantic features, as another embodiment, based on the trained semantic retrieval model, extracting semantic features of the image to be retrieved may include:

[0090] Obtain the second encoder of the semantic retrieval model obtained by jointly training multiple loss functions.

[0091] The image to be retrieved is input into the second encoder, which is then used to calculate the semantic features of the image to be retrieved.

[0092] A semantic retrieval model may include a second encoder and a target training task. The target training task can be a training task consisting of loss calculations using multiple loss functions.

[0093] In this embodiment, a second encoder using a semantic retrieval model with multiple training tasks is employed to accurately extract features from the image to be retrieved, thereby improving the efficiency and accuracy of feature extraction.

[0094] As an example, in order to obtain an accurate target image, such as Figure 3 The diagram shown is yet another flowchart of an image retrieval method provided in this disclosure. This method can be configured as an image retrieval device, which can be installed in an electronic device. Figure 2 The difference in the illustrated embodiment is that step 205 above: based on target features, searching for target images in the image library that meet the image similarity criteria to the image to be retrieved may include:

[0095] 301: Identify multiple candidate images in the image library.

[0096] The feature extraction steps for multiple candidate images in the image library are the same as the feature extraction steps for the image to be retrieved.

[0097] 302: Extract candidate visual features corresponding to multiple candidate images based on a visual retrieval model.

[0098] Optionally, extracting candidate visual features corresponding to multiple candidate images based on a visual retrieval model may include: obtaining a first encoder of a visual retrieval model obtained by joint training of multiple training tasks; inputting candidate images into the first encoder and using the first encoder to calculate candidate visual features of the image to be retrieved, thereby obtaining candidate visual features corresponding to multiple candidate images.

[0099] 303: Extract candidate semantic features corresponding to multiple candidate images based on a semantic retrieval model.

[0100] Optionally, extracting candidate semantic features corresponding to multiple candidate images based on the semantic retrieval model may include: obtaining a second encoder of the semantic retrieval model obtained by joint training of multiple loss functions; inputting the candidate images into the second encoder, and using the second encoder to calculate the candidate semantic features of the image to be retrieved, so as to obtain the candidate semantic features corresponding to multiple candidate images respectively.

[0101] 304: Concatenate the candidate visual features and candidate semantic features of each candidate image to obtain the candidate features corresponding to multiple candidate images.

[0102] By concatenating the candidate visual features and candidate semantic features of a candidate image, we can obtain the candidate features of that candidate image, and thus obtain the candidate features corresponding to multiple candidate images.

[0103] 305: Based on the candidate features corresponding to multiple candidate images, find the target feature that satisfies the feature similarity condition with the target feature, and obtain the candidate image corresponding to the target feature as the target image.

[0104] In this embodiment, multiple candidate images are identified in an image library. Visual and semantic retrieval models are used to extract visual and semantic features from each candidate image, obtaining candidate visual features and semantic visual features corresponding to each candidate image. Candidate features for each candidate image are obtained through feature concatenation. By extracting these candidate features, the same feature extraction method can be used on the candidate images in the image library as on the image to be retrieved, achieving accurate feature extraction and enabling accurate matching.

[0105] In one possible design, step 305 above, based on the candidate features corresponding to multiple candidate images, the target features whose differences satisfy the feature similarity condition with the target features may include:

[0106] Spatial clustering is performed on the candidate features corresponding to multiple candidate images to obtain a spatial tree formed by multiple spatial nodes, the image category corresponding to the spatial node, and the spatial features; the image category of the parent spatial node is the parent category of the image category of the child spatial node; the image category corresponding to each spatial node includes at least one candidate feature corresponding to the candidate image.

[0107] Based on the spatial features corresponding to multiple spatial nodes in the spatial tree, target spatial nodes that satisfy the spatial similarity condition with the target features are identified.

[0108] From the candidate features corresponding to at least one candidate image of the target spatial node, determine the target features that satisfy the feature similarity condition with the target features.

[0109] Optionally, spatial clustering of candidate features corresponding to multiple candidate images may include using the Generalized Non-Orthogonal Inverted Multi-Index (GNOIMI) retrieval algorithm.

[0110] In this embodiment, spatial clustering is performed on the candidate features corresponding to multiple candidate images to obtain a spatial tree formed by multiple spatial nodes. Each spatial node corresponds to an image category and a spatial feature. The image category of a parent spatial node in the spatial tree is the parent category of the image categories of its child spatial nodes. Each spatial node's corresponding image category includes at least one candidate feature corresponding to a candidate image. Through spatial category clustering, images in the image library can be effectively classified, grouping highly similar images into a single space, thus achieving accurate clustering of image features. By utilizing the spatial features corresponding to multiple spatial nodes in the spatial tree, target spatial nodes that satisfy the spatial similarity condition with the target feature can be determined, enabling spatial node selection. The selection of spatial nodes has a hierarchical selection characteristic, which can accelerate the determination of target spatial nodes. By sequentially matching the candidate features corresponding to at least one candidate image of the target spatial node with the target feature for feature similarity, the target feature that satisfies the feature similarity condition is finally obtained, improving the efficiency and accuracy of target feature selection.

[0111] As one example, based on the spatial features corresponding to multiple spatial nodes in the spatial tree, target spatial nodes that satisfy the spatial similarity condition with the target features are determined, including:

[0112] Starting from the root node of the spatial tree, calculate the feature distance between the spatial features of each spatial node and the target features in turn;

[0113] If the feature distance of any spatial node is determined to be less than the distance threshold, then the spatial node is determined to be a target spatial node that satisfies the spatial similarity condition with the target feature.

[0114] Optionally, the feature distance between the spatial features of any spatial node and the target feature can be calculated. If the feature distance is less than a distance threshold, at least one sub-space node of that spatial node is obtained. If it is determined that at least one sub-space node is not empty, the feature sub-distances between the spatial features corresponding to each of the at least one sub-space node and the target feature are calculated to obtain the feature sub-distances corresponding to each of the at least one sub-space node. If any feature sub-distance is greater than a distance sub-threshold, the sub-space node corresponding to that feature sub-distance is determined as a new spatial node, and the step of determining the feature distance between the spatial features of any spatial node and the target feature is continued until at least one sub-space node is empty.

[0115] Among them, the distance sub-threshold is greater than the distance threshold. The distance sub-threshold can be calculated based on the distance threshold and the threshold increment, or it can be a value greater than the distance threshold that is randomly determined based on the distance threshold.

[0116] Optionally, if at least one sub-space node of a spatial node is determined to be empty, then the spatial node is determined to be the target spatial node. If the feature distance of any spatial node is determined to be greater than or equal to a distance threshold, then the spatial node is determined not to satisfy the spatial similarity condition.

[0117] In this embodiment of the disclosure, when using a spatial tree to find target spatial nodes, the feature distance between the spatial features of each spatial node and the target feature can be calculated sequentially, starting from the root node of the spatial tree. If the feature distance of any spatial node is less than a distance threshold, the spatial node is determined to be a target spatial node that satisfies the spatial similarity condition with the target feature. By matching spatial features with feature distances, the target feature can be accurately matched with each spatial node, thus achieving accurate finding of the target spatial node.

[0118] like Figure 4 The diagram shown is a flowchart of a visual retrieval model training method provided in this embodiment. This method can be configured as a visual retrieval model training device, which can be located in an electronic device. The visual retrieval model training method may include:

[0119] 401: Based on the visual similarity clustering algorithm, multiple candidate images in the image library are visually clustered to obtain at least one visual category.

[0120] Optionally, the visual similarity clustering algorithm may include a semi-supervised clustering algorithm. Semi-supervised clustering algorithms can achieve faster and more accurate distance analysis of multiple candidate images. Based on the visual similarity clustering algorithm, multiple candidate images in an image library are visually clustered to obtain at least one visual category. This may include: extracting visual features from multiple candidate images in the image library to obtain candidate visual features corresponding to each candidate image; dividing the candidate visual features according to semantic similarity; and grouping candidate visual features that meet the semantic similarity condition into the same visual category to obtain at least one visual category. The number of at least one visual category is relatively large, for example, it may include more than 80,000, with over 80,000 visual categories existing. A large number of visual categories allows for more detailed and accurate visual category division of multiple candidate images in the image library, enabling effective management of the visual characteristics of images in the image library.

[0121] Each visual category may include at least one visually similar candidate image.

[0122] Each visual category can correspond to a specific visual vector or visual label to distinguish different visual categories. The visual vector or visual label for any visual category can be obtained by extracting the common features from at least one candidate visual feature corresponding to that visual category. For example, the mobile phone brand M can include two models, M1 and M2, which have different appearances. Both M1 and M2 can be considered as a visual category. Candidate images corresponding to the M1 model can be classified into the same visual category, and candidate images corresponding to the M2 model can be classified into the same visual category. In this case, the visual vector for the M1 model is "the feature vector corresponding to the appearance of the M1 model phone," or the visual label is "M brand M1 model phone." The visual vector for the M2 model is "the feature vector corresponding to the appearance of the M2 model phone," or the visual label is "M brand M1 model phone." Visual vectors can identify the semantic characteristics of a category.

[0123] 402: Generate a visual retrieval model to be trained, which includes a first encoder and multiple training tasks.

[0124] Multiple training tasks include: a classification task corresponding to at least one visual category and a similarity measurement task.

[0125] 403: Determine at least one first training data; the first training data includes a first training image labeled with a visual ground truth and a second training image.

[0126] The visual ground truth includes a visual category ground truth determined based on at least one visual category and a similarity metric ground truth. The visual category ground truth is the true visual category of the first training image and the second training image. The similarity metric ground truth is the similarity measurement result of the first training image and the second training image.

[0127] 404: Input at least one first training data into the visual retrieval model to be trained, and take the first loss value generated by multiple training tasks on the overall visual retrieval model satisfying the first error condition as the training objective, and train to obtain the visual model parameters of the visual retrieval model.

[0128] In this model, the first encoder of the visual retrieval model, corresponding to the visual model parameters, is used to extract the visual features of the image to be retrieved. These visual features are then used to find the target image from the image database.

[0129] The visual feature and semantic retrieval model extracts semantic features from the image to be retrieved and uses them to find the target image from the image database; the semantic retrieval model is obtained by joint training based on multiple loss functions.

[0130] Optionally, after obtaining the visual model parameters of the visual retrieval model, the visual model parameters of the visual retrieval model can be stored in a visual parameter file so that when the visual retrieval model is needed, the visual model parameters can be read from the visual parameter file and loaded based on the visual retrieval model to complete the acquisition of the visual retrieval model.

[0131] The first encoder of the visual retrieval model corresponding to the visual model parameters is also used to extract candidate visual features corresponding to multiple candidate images in the image library.

[0132] In this embodiment, a visual similarity clustering algorithm is used to perform visual clustering processing on multiple candidate images in an image library to obtain at least one visual category. Each visual category contains at least one candidate image that satisfies the visual similarity requirement. Extracting at least one visual category ensures that the image library is divided into visual categories. Then, multiple training tasks, consisting of a classification task determined by at least one visual category and a similarity measurement task, are used to train the visual retrieval model. This ensures that the first encoder in the visual retrieval model is a machine model that extracts the visual features of the images. Through training the visual retrieval model, accurate visual model parameters are obtained, enabling the first encoder in the trained visual retrieval model to have strong feature extraction performance for at least one visual category, thereby improving the accuracy of the first encoder in extracting visual features.

[0133] As one example, the training tasks include a classification task and a similarity measurement task. Generating a visual retrieval model to be trained, which includes a first encoder and multiple training tasks, may include:

[0134] The classification task involves constructing a visual retrieval model based on at least one visual category.

[0135] Based on the similarity measurement model, a similarity measurement task is constructed for the visual retrieval model to obtain multiple training tasks corresponding to the classification task and the similarity measurement task.

[0136] The first encoder of the visual retrieval model is constructed based on a neural network algorithm;

[0137] The first encoder and the visual retrieval model to be trained, consisting of multiple training tasks, are determined.

[0138] Optionally, the neural network algorithm may include a deep neural network algorithm or a convolutional neural network algorithm. The first encoder may employ a multi-level network structure to accurately define the first encoder.

[0139] The classification task in multiple training tasks may include a fully connected layer, which can be generated from at least one visual category. The generation method of the fully connected layer can be found in the description of relevant techniques and will not be repeated here. The similarity measurement task in multiple training tasks may also include a fully connected layer, which can be used to perform pixel-level measurements on a first visual feature of a first training image and a second visual feature of a second training image to obtain the similarity between the first and second training images.

[0140] In this embodiment of the disclosure, a classification task can be constructed based on at least one visual category, and a similarity measurement task can be constructed based on at least one similarity measurement model. Multiple training tasks can be obtained by constructing the classification task and the similarity measurement task. A first encoder of the visual retrieval model can be constructed using a neural network algorithm. By using at least one visual category as the basis for constructing the classification task, the first encoder of the visual retrieval model can distinguish between encoders of different categories, resulting in a first encoder with more accurate visual feature extraction capabilities.

[0141] As one embodiment, at least one first training data is input into the visual retrieval model to be trained, and the training objective is to obtain the visual model parameters of the visual retrieval model by training the first loss value generated by multiple training tasks on the overall visual retrieval model to meet a first error condition, including:

[0142] Initialize the parameters of the visual retrieval model to be trained to obtain the first model parameters corresponding to the visual model parameters;

[0143] At least one first training data is sequentially input into the first encoder of the visual retrieval model corresponding to the first model parameters, and the first training features corresponding to at least one first training data are extracted respectively.

[0144] Based on the first training features and visual ground values ​​corresponding to at least one first training data, calculate the classification error of at least one first training data in the classification task and the similarity measurement error in the similarity measurement task.

[0145] The classification error and the similarity metric error are weighted together to obtain the first loss value generated during training;

[0146] If the first loss value is determined to satisfy the first error condition, then the first model parameter is determined to be the visual model parameter of the visual retrieval model.

[0147] The first training data may include a first training image and a second training image. The first training data is input into the visual retrieval model corresponding to the first model parameters. The first retrieval model can then be used to extract visual features from the first and second training images respectively, obtaining a first visual feature of the first training image and a second visual feature of the second training image. This results in a first training feature composed of the first and second visual features.

[0148] In this embodiment, at least one first training data point is sequentially input into a first encoder corresponding to the first model parameters. The first encoder extracts the first training features corresponding to each of the at least one first training data point, thus achieving accurate extraction of the first training features. Using the first training features and the visual ground truth of each first training data point, the classification efficiency of the visual retrieval model for at least one first training data point in the classification task and the measurement results in the similarity measurement task can be calculated to obtain the classification error and the similarity measurement error. The first loss value corresponding to the classification error and the similarity measurement error can be used to accurately determine whether the first model parameters meet the second error condition. This completes the accurate training of the visual retrieval model.

[0149] As another embodiment, after weighting the classification error and the similarity measurement error to obtain the first loss value generated during training, the method further includes:

[0150] If it is determined that the first loss value does not meet the first error condition, the first model parameters of the visual retrieval model are updated, and the process returns to the step of sequentially inputting at least one first training data into the first encoder of the visual retrieval model corresponding to the first model parameters to continue execution.

[0151] The first loss value satisfies the first error condition if it is less than the first error threshold. The first loss value does not satisfy the first error condition if it is greater than or equal to the first error threshold. The first error threshold can be set according to actual usage requirements.

[0152] In this embodiment of the disclosure, after obtaining the first loss value by weighting the classification error and the similarity measurement error, the first model parameters of the visual retrieval model can be updated if it is determined that the first loss value does not meet the first error condition. By continuously updating and iterating the model parameters of the visual retrieval model, effective and accurate training of the visual retrieval model can be achieved.

[0153] In one possible design, based on the first training features and visual ground truth corresponding to at least one first training data, the classification error of at least one first training data in the classification task and the similarity measurement error in the similarity measurement task are calculated, including:

[0154] The first training feature of the first training data is input into the classification task to obtain at least one classification result corresponding to the first training data; the classification result is the target visual category determined from at least one visual category;

[0155] The classification error is calculated based on the target visual category and the true value of the visual category corresponding to at least one first training data.

[0156] Input the first training feature of the first training data into the similarity measurement task to obtain the similarity measurement result of the similarity measurement task on the first training image and the second training image in the first training data, so as to obtain the similarity measurement result corresponding to at least one training data respectively.

[0157] The similarity measurement error is calculated based on the similarity measurement results and the true value of the similarity measurement corresponding to at least one first training data.

[0158] Optionally, calculating the classification error based on the target visual category and ground truth value corresponding to at least one first training data point may include: calculating the visual distance between the target visual category and ground truth value of any first training data point, obtaining the visual distance corresponding to at least one first training data point, and calculating the classification error based on the visual distance corresponding to at least one first training data point and a visual loss function. Calculating the similarity measurement error based on the similarity measurement result and ground truth value corresponding to at least one first training data point may include: calculating the similarity error between the similarity measurement result and ground truth value of any first training data point, obtaining the similarity error corresponding to at least one first training data point, and calculating the similarity measurement error based on the similarity error corresponding to at least one first training data point and a similarity loss function.

[0159] Among them, the visual loss function may include, for example, the cross-entropy loss function, and the similarity loss function may include, for example, the lifted loss function.

[0160] In this embodiment of the disclosure, multiple training tasks are performed on the first training features corresponding to at least one first training data. Each first training feature can be input into a classification task to obtain a classification result corresponding to at least one training data. The classification result is the target visual category determined from at least one visual category, thus completing the classification task. Each feature can be input into a similarity measurement task to measure the similarity between two visual features in the first training features, thus completing the image similarity measurement task. Through similarity measurement and visual category classification, accurate error detection can be performed on the similarity between the first training image and the second training image of the first training data, as well as their visual categories, obtaining accurate classification error and similarity measurement error.

[0161] like Figure 5 The diagram shown is a flowchart of a semantic retrieval model training method provided in this embodiment. This method can be configured as a semantic retrieval model training device, which can be located in an electronic device. The semantic retrieval model training method may include:

[0162] 501: Based on the semantic category mining algorithm, semantic category mining is performed on multiple candidate images in the image library to obtain at least one semantic category.

[0163] Optionally, based on a semantic category mining algorithm, multiple candidate images in the image library are subjected to speech category mining to obtain at least one semantic category. This can include: extracting semantic features from multiple candidate images in the image library to obtain candidate semantic features corresponding to each candidate image; classifying the candidate semantic features corresponding to each candidate image according to semantic similarity; and assigning candidate semantic features that meet the semantic similarity condition to the same semantic category to obtain at least one semantic category. The number of at least one semantic category is relatively large, for example, higher than 100,000, meaning there are more than 100,000 semantic categories. With a large number of semantic categories, more detailed and accurate semantic category classification can be performed on multiple candidate images in the image library, achieving effective management of the semantic characteristics of images in the image library.

[0164] Each semantic category can correspond to a semantic vector or semantic label to distinguish different semantic categories. The semantic vector or semantic label of any semantic category can be obtained by extracting the common features from at least one candidate semantic feature corresponding to that semantic category. For example, mobile phones can include brand A and brand B. Candidate images corresponding to brand A mobile phones can be classified into the same semantic category, and candidate images corresponding to brand B mobile phones can be classified into the same semantic category. In this case, the semantic vector of brand A mobile phones is "the word vector corresponding to brand A mobile phones" or the semantic label is "brand A mobile phones". The semantic vector of brand B mobile phones is "the word vector corresponding to brand B mobile phones" or the semantic label is "brand B mobile phones". Semantic vectors can identify the meaning characteristics of semantic categories.

[0165] 502: Generate a semantic retrieval model to be trained, which includes a second encoder and a target training task; the target training task corresponds to multiple loss functions.

[0166] The target training task may include a semantic classification task corresponding to at least one semantic category. The semantic classification task can determine the corresponding target semantic category from at least one semantic category for the training features to be classified.

[0167] 503: Determine at least one second training data point. The second training data includes a third training image labeled with semantic category ground truth values.

[0168] The third training image can be any type of image. To improve image acquisition efficiency, the third training image can be obtained from an image library.

[0169] 504: Input at least one second training data into the semantic retrieval model to be trained, and use the second loss value obtained by calculating the overall loss of the semantic retrieval model based on multiple loss functions to satisfy the second error condition as the training objective, and train to obtain the semantic model parameters of the semantic retrieval model.

[0170] The second encoder in the semantic retrieval model, corresponding to the semantic model parameters, is used to extract the semantic features of the image to be retrieved; the semantic features are used to find the target image from the image library.

[0171] The semantic features and visual retrieval model extract visual features from the image to be retrieved and use them to find the target image from the image database; the visual retrieval model is obtained by joint training based on multiple training tasks.

[0172] Optionally, after obtaining the semantic model parameters of the semantic retrieval model, the semantic model parameters of the semantic retrieval model can be stored in a semantic parameter file so that when the semantic retrieval model is needed, the semantic model parameters can be read from the semantic parameter file and loaded based on the semantic retrieval model to complete the acquisition of the semantic retrieval model.

[0173] In this embodiment, a visual similarity mining algorithm is used to mine semantic categories from multiple candidate images in an image library, thereby obtaining at least one semantic category. Each semantic category contains at least one candidate image that satisfies semantic similarity. Extracting at least one semantic category ensures that the image library is divided according to semantics, enriching the semantic categories of the images. A target training task is then constructed using at least one semantic category; this target training task is a semantic classification task containing at least one semantic category. During training, the training target of the third training image and its corresponding semantic category ground truth can be used as the training objective to train and obtain the semantic model parameters of the semantic retrieval model. Training through the target training task ensures that the semantic retrieval model can have a certain feature discrimination for at least one semantic category, enabling the features extracted by the second encoder of the semantic retrieval model to distinguish different semantic categories, thus achieving accurate training of the semantic retrieval model.

[0174] The second encoder in the semantic retrieval model corresponding to the semantic retrieval parameters is also used to extract candidate semantic features of candidate images in the image library.

[0175] As one embodiment, generating a semantic retrieval model to be trained, including a second encoder and a target training task, includes:

[0176] The objective training task of constructing a semantic retrieval model based on at least one semantic category; the objective training task corresponds to multiple loss functions.

[0177] A second encoder for a semantic retrieval model is constructed based on a neural network algorithm.

[0178] Determine the semantic retrieval model consisting of the second encoder and the target training task.

[0179] Optionally, the neural network algorithm may include a deep neural network algorithm or a convolutional neural network algorithm. The second encoder may employ a multi-level network structure to accurately define the second encoder.

[0180] The target training task may include fully connected layers, which can be generated from at least one semantic category. The generation methods for fully connected layers can be found in the descriptions of relevant technologies, and will not be elaborated upon here.

[0181] In this embodiment, a target training task for constructing a semantic retrieval model is built using at least one semantic category. Then, a second encoder for the semantic retrieval model is constructed based on a neural network algorithm. Finally, a semantic retrieval model consisting of the second encoder and the target training task is obtained. The target training task is a semantic classification task for at least one semantic category. The model is trained using the second encoder and the semantic retrieval model corresponding to the target training task, achieving effective management of semantic categories and the second encoder, and obtaining a highly accurate semantic retrieval model.

[0182] As one possible implementation, at least one second training data is input into the semantic retrieval model to be trained. The training objective is to obtain the semantic model parameters of the semantic retrieval model by training a second loss value obtained from the overall loss calculation of the semantic retrieval model based on multiple loss functions, which satisfies a second error condition. These parameters include:

[0183] Initialize the parameters of the semantic retrieval model to be trained to obtain the second model parameters corresponding to the semantic model parameters;

[0184] At least one second training data is sequentially input into the second encoder of the semantic retrieval model corresponding to the second model parameters, and the second training features corresponding to at least one second training data are extracted respectively.

[0185] Based on the second training features and semantic category ground values ​​corresponding to at least one second training data, calculate the semantic recognition error and center loss error of at least one second training data in the target training task.

[0186] The semantic recognition error and the center loss error are weighted together to obtain the second loss value generated during training;

[0187] If the second loss value is determined to satisfy the second error condition, then the second model parameter is determined to be the semantic model parameter of the semantic retrieval model.

[0188] Optionally, if the second model parameters are determined to be the semantic model parameters of the semantic retrieval model, then training of the semantic retrieval model can be stopped.

[0189] By sequentially inputting at least one second training data into the second encoder of the semantic retrieval model corresponding to the second model parameters, the second encoder can be used to extract features from each second training data to obtain the second training features corresponding to the second encoder.

[0190] The second loss value satisfies the second error condition if it is less than the second error threshold. The second loss value does not satisfy the second error condition if it is greater than or equal to the second error threshold. The second error threshold can be set according to actual usage requirements.

[0191] In this embodiment of the disclosure, after obtaining the second model parameters of the semantic model, at least one second training data can be sequentially input into the second encoder of the semantic retrieval model corresponding to the second model parameters. The second encoder extracts the second training features corresponding to each of the at least one second training data, enabling accurate extraction of the second training features. Using the second training features and the semantic category ground truth of each second training data, the semantic recognition error and center loss error can be calculated. The semantic recognition error and center loss error can be used to accurately determine whether the second model parameters satisfy the second error condition, thus completing the accurate training of the semantic retrieval model.

[0192] As another embodiment, if it is determined that the second loss value does not meet the second error condition, the second model parameters of the semantic retrieval model are updated, and the process returns to the step of sequentially inputting at least one second training data into the second encoder of the semantic retrieval model corresponding to the second model parameters to continue execution.

[0193] In this embodiment of the disclosure, after obtaining the second loss value by weighting the semantic recognition error and the center loss error, if it is determined that the second loss value does not meet the second error condition, the second model parameters of the semantic retrieval model can be updated. By continuously updating and iterating the model parameters of the semantic retrieval model, effective and accurate training of the semantic retrieval model can be achieved.

[0194] In some embodiments, the target training task includes a semantic classification task based on at least one semantic category; and, based on the second training features and semantic category ground truth values ​​corresponding to at least one second training data, the semantic recognition error and center loss error of at least one second training data in the target training task are calculated, including:

[0195] Input the second training features of the second training data into the semantic classification task corresponding to the target training task to obtain the target semantic category corresponding to the second training data, so as to obtain at least one target semantic category corresponding to the second training data respectively.

[0196] Based on the error loss function, the error is calculated for the target semantic category and the semantic category ground value corresponding to at least one second training data respectively, to obtain the semantic recognition error;

[0197] Based on the center loss function, the error is calculated for the target semantic category and the semantic category ground truth value corresponding to at least one second training data to obtain the center loss error.

[0198] Optionally, the second training feature is input into the semantic classification task. The semantic classification task can calculate the feature distance between the second training feature and the semantic vector or semantic label corresponding to at least one semantic category. The semantic category corresponding to the minimum feature distance among the feature distances corresponding to at least one semantic category is determined as the target semantic category of the second training feature.

[0199] As an optional approach, the semantic recognition error can be obtained by using an error loss function to calculate the error between the target semantic category and the semantic category ground truth value corresponding to at least one second training data. This can include: calculating the error between the target semantic category and the semantic category ground truth value in any second training data to obtain the error loss corresponding to at least one second training data, and adding the error losses corresponding to at least one second training data to obtain the semantic recognition error.

[0200] As an optional approach, the center loss function is used to calculate the error of the target semantic category and the true value of the semantic category corresponding to at least one second training data to obtain the center loss error. This can include: inputting the target semantic category and the true value of the semantic category corresponding to at least one second training data into the center loss function, and using the center loss function to calculate the center loss error.

[0201] The error loss function can include cross-entropy, absolute value loss, logarithmic loss, etc. This embodiment does not impose excessive restrictions on the specific type of error loss function. The center loss function can refer to using clustering of features and feature centers, along with softmax loss (regression loss), as the loss function to reduce intra-class distances. The center loss function and the error loss function can be used to accurately calculate the overall loss.

[0202] In this embodiment of the disclosure, the target training task is specifically a semantic classification task based on at least one semantic category. By inputting the second training features of each second training data point into the semantic classification task, the target semantic category of the second training features can be obtained. By inputting the second training features into the semantic classification task, training classification of at least one second training data point can be achieved. Having obtained the target semantic categories corresponding to at least one second training data point, the semantic recognition error can be calculated based on an error loss function, and the center loss error can be calculated based on a center loss function. Through the calculation of these two loss functions, the training error of at least one second training data point can be accurately calculated, resulting in accurate semantic recognition error and center loss error.

[0203] like Figure 6 The diagram shown is a structural schematic of an image retrieval device provided in an embodiment of this disclosure. This device can apply the image retrieval method described above and can be installed in an electronic device. The image retrieval device 600 may include the following units:

[0204] Image acquisition unit 601: used to acquire the image to be retrieved;

[0205] First extraction unit 602: used to extract visual features of the image to be retrieved from the visual retrieval model obtained by joint training of multiple training tasks;

[0206] The second extraction unit 603 is used to extract the semantic features of the image to be retrieved based on the semantic retrieval model obtained by joint training of multiple loss functions.

[0207] First splicing unit 604: used to splice visual features and semantic features to obtain target features of the image to be retrieved;

[0208] Target search unit 605: Used to search for target images in the image library that meet the image similarity conditions to the image to be searched, based on target features.

[0209] In this embodiment, after obtaining the image to be retrieved, multiple training tasks can be used in conjunction with a visual retrieval model to extract visual features of the image to be retrieved, and multiple training tasks can be used in conjunction with a semantic retrieval model to extract semantic features of the image to be retrieved. The visual and semantic features are then concatenated to obtain target features. These target features can be used to find target images in the image library that meet the image similarity criteria to the image to be retrieved. The semantic and visual features integrate both visual and semantic aspects of the image to be retrieved, resulting in more comprehensive features and enabling accurate identification of the image to be retrieved.

[0210] As one embodiment, the image acquisition unit includes:

[0211] The request receiving module is used to receive image retrieval requests sent by user equipment;

[0212] The image acquisition module is used to acquire the image to be retrieved in response to an image retrieval request.

[0213] In some embodiments, the first extraction unit includes:

[0214] The first acquisition module is used to acquire the first encoder of the visual retrieval model obtained by joint training of multiple training tasks.

[0215] The first input module is used to input the image to be retrieved into the first encoder and use the first encoder to calculate the visual features of the image to be retrieved.

[0216] As one possible implementation, the second extraction unit includes:

[0217] The second acquisition module is used to acquire the second encoder of the semantic retrieval model obtained by joint training of multiple loss functions;

[0218] The second input module is used to input the image to be retrieved into the second encoder, and to use the second encoder to calculate the semantic features of the image to be retrieved.

[0219] In some embodiments, the target finding unit includes:

[0220] The candidate determination module is used to determine multiple candidate images from the image library;

[0221] The first extraction module is used to extract candidate visual features corresponding to multiple candidate images based on a visual retrieval model.

[0222] The second extraction module is used to extract candidate semantic features corresponding to multiple candidate images based on the semantic retrieval model.

[0223] The first stitching module is used to stitch together the candidate visual features and candidate semantic features of each candidate image to obtain the candidate features corresponding to multiple candidate images respectively.

[0224] The feature matching module is used to find target features that meet the feature similarity condition with the target features based on the candidate features corresponding to multiple candidate images, so as to obtain the candidate image corresponding to the target feature as the target image.

[0225] In some possible designs, the feature matching module includes:

[0226] The spatial clustering submodule is used to perform spatial clustering on the candidate features corresponding to multiple candidate images to obtain a spatial tree formed by multiple spatial nodes, the image category corresponding to the spatial node, and spatial features; the image category of the parent spatial node is the parent category of the image categories of its child spatial nodes; the image category corresponding to each spatial node includes at least one candidate feature corresponding to the candidate image.

[0227] The spatial matching submodule is used to determine the target spatial node that satisfies the spatial similarity condition with the target feature based on the spatial features corresponding to multiple spatial nodes in the spatial tree.

[0228] The feature determination submodule is used to determine the target features that satisfy the feature similarity condition from the candidate features corresponding to at least one candidate image of the target spatial node.

[0229] In some embodiments, the spatial matching submodule is specifically used for:

[0230] Starting from the root node of the spatial tree, calculate the feature distance between the spatial features of each spatial node and the target feature in turn; if the feature distance of any spatial node is determined to be less than the distance threshold, then the spatial node is determined to be the target spatial node that satisfies the spatial similarity condition with the target feature.

[0231] like Figure 7 The diagram shown is a structural schematic of a visual retrieval model training device provided in an embodiment of this disclosure. This device can apply the visual retrieval model training method described in the above embodiments, and the visual retrieval model training device can be installed in an electronic device. The visual retrieval model training device 700 may include the following units:

[0232] Category clustering unit 701: Used for visual clustering algorithms based on visual similarity to perform visual clustering processing on multiple candidate images in the image library to obtain at least one visual category;

[0233] First generation unit 702: used to generate a visual retrieval model to be trained, which includes a first encoder and multiple training tasks;

[0234] First determining unit 703: used to determine at least one first training data; the first training data includes a first training image labeled with a visual ground truth and a second training image;

[0235] First training unit 704: used to input at least one first training data into the visual retrieval model to be trained, with the first loss value generated by multiple training tasks on the overall visual retrieval model satisfying the first error condition as the training objective, and to train to obtain the visual model parameters of the visual retrieval model.

[0236] Among them, the first encoder of the visual retrieval model corresponding to the visual model parameters is used to extract the visual features of the image to be retrieved; the visual features and the semantic features extracted by the semantic retrieval model from the image to be retrieved are used to find the target image from the image library; the semantic retrieval model is obtained by joint training based on multiple loss functions.

[0237] In this embodiment, a visual similarity clustering algorithm is used to perform visual clustering processing on multiple candidate images in an image library to obtain at least one visual category. Each visual category contains at least one candidate image that satisfies the visual similarity requirement. Extracting at least one visual category ensures that the image library is divided into visual categories. Then, multiple training tasks, consisting of a classification task determined by at least one visual category and a similarity measurement task, are used to train the visual retrieval model. This ensures that the first encoder in the visual retrieval model is a machine model that extracts the visual features of the images. Through training the visual retrieval model, accurate visual model parameters are obtained, enabling the first encoder in the trained visual retrieval model to have strong feature extraction performance for at least one visual category, thereby improving the accuracy of the first encoder in extracting visual features.

[0238] As one embodiment, the training tasks include: a classification task and a similarity measurement task; the first generation unit includes:

[0239] The first building module is used for the classification task of building a visual retrieval model based on at least one visual category;

[0240] The second building module is used to build a similarity measurement task for the visual retrieval model based on the similarity measurement model, so as to obtain multiple training tasks corresponding to the classification task and the similarity measurement task.

[0241] The first encoding module is used to construct the first encoder of the visual retrieval model based on the neural network algorithm;

[0242] The first determining module is used to determine the visual retrieval model to be trained, which consists of the first encoder and multiple training tasks.

[0243] In one possible design, the first training unit includes:

[0244] The first initialization module is used to initialize the parameters of the visual retrieval model to be trained and obtain the first model parameters corresponding to the visual model parameters.

[0245] The first training module is used to sequentially input at least one first training data into the first encoder of the visual retrieval model corresponding to the first model parameters, and extract the first training features corresponding to at least one first training data respectively.

[0246] The first error module is used to calculate the classification error of at least one first training data in the classification task and the similarity measurement error in the similarity measurement task based on the first training features and visual ground values ​​corresponding to at least one first training data respectively.

[0247] The first weighting module is used to weight the classification error and the similarity measurement error to obtain the first loss value generated during training.

[0248] The first determining module is used to determine the first model parameters as the visual model parameters of the visual retrieval model if the first loss value satisfies the first error condition.

[0249] As an optional approach, the first training unit also includes

[0250] The second determining module is used to update the first model parameters of the visual retrieval model if it is determined that the first loss value does not meet the first error condition, and return to the step of sequentially inputting at least one first training data into the first encoder of the visual retrieval model corresponding to the first model parameters to continue execution.

[0251] In some embodiments, the first error module includes:

[0252] The first training submodule is used to input the first training features of the first training data into the classification task to obtain at least one classification result corresponding to the first training data; the classification result is the target visual category determined from at least one visual category.

[0253] The classification error submodule is used to calculate the classification error based on the target visual category and the true value of the visual category corresponding to at least one first training data.

[0254] The similarity measurement submodule is used to input the first training features of the first training data into the similarity measurement task to obtain the similarity measurement results of the similarity measurement task on the first training image and the second training image in the first training data, so as to obtain the similarity measurement results corresponding to at least one training data respectively.

[0255] The error calculation submodule is used to calculate the similarity measurement error based on the similarity measurement result and the true value of the similarity measurement corresponding to at least one first training data.

[0256] like Figure 8 The diagram shown is a structural schematic of a semantic retrieval model training device provided in an embodiment of this disclosure. This device can apply the semantic retrieval model training method described in the above embodiments, and the semantic retrieval model training device can be installed in an electronic device. The semantic retrieval model training device 800 may include the following units:

[0257] Category mining unit 801: Used to perform semantic category mining on multiple candidate images in the image library based on the semantic category mining algorithm to obtain at least one semantic category;

[0258] Second generation unit 802: used to generate a semantic retrieval model to be trained, which includes a second encoder and a target training task; the target training task corresponds to multiple loss functions;

[0259] Second determining unit 803: for determining at least one second training data; the second training data includes a third training image labeled with semantic category ground truth values;

[0260] The second training unit 804 is used to input at least one second training data into the semantic retrieval model to be trained, with the second loss value obtained by calculating the overall loss of the semantic retrieval model based on multiple loss functions satisfying the second error condition as the training objective, and to train and obtain the semantic model parameters of the semantic retrieval model.

[0261] Among them, the second encoder in the semantic retrieval model corresponding to the semantic model parameters is used to extract the semantic features of the image to be retrieved; the semantic features and the visual features extracted by the visual retrieval model from the image to be retrieved are used to find the target image from the image library; the visual retrieval model is obtained by joint training based on multiple training tasks.

[0262] In this embodiment, a visual similarity mining algorithm is used to mine semantic categories from multiple candidate images in an image library, thereby obtaining at least one semantic category. Each semantic category contains at least one candidate image that satisfies semantic similarity. Extracting at least one semantic category ensures that the image library is divided according to semantics, enriching the semantic categories of the images. A target training task is then constructed using at least one semantic category; this target training task is a semantic classification task containing at least one semantic category. During training, the training target of the third training image and its corresponding semantic category ground truth can be used as the training objective to train and obtain the semantic model parameters of the semantic retrieval model. Training through the target training task ensures that the semantic retrieval model can have a certain feature discrimination for at least one semantic category, enabling the features extracted by the second encoder of the semantic retrieval model to distinguish different semantic categories, thus achieving accurate training of the semantic retrieval model.

[0263] As one embodiment, the second generation unit includes:

[0264] The objective training task of constructing a semantic retrieval model is based on at least one semantic category; the objective training task corresponds to multiple loss functions.

[0265] A second encoder for the semantic retrieval model is constructed based on a neural network algorithm.

[0266] Determine the semantic retrieval model consisting of the second encoder and the target training task.

[0267] In some embodiments, the second training unit includes:

[0268] The second initialization module is used to initialize the parameters of the semantic retrieval model to be trained, and obtain the second model parameters corresponding to the semantic model parameters.

[0269] The second training module is used to sequentially input at least one second training data into the second encoder of the semantic retrieval model corresponding to the second model parameters, and extract the second training features corresponding to at least one second training data respectively.

[0270] The second error module is used to calculate the semantic recognition error and center loss error of at least one second training data in the target training task based on the second training features corresponding to at least one second training data respectively.

[0271] The second weighting module is used to weight the semantic recognition error and the center loss error to obtain the second loss value generated during training.

[0272] The second determining module is used to determine the second model parameters as the semantic model parameters of the semantic retrieval model if the second loss value satisfies the second error condition.

[0273] In one possible design, the target training task includes a semantic classification task based on at least one semantic category; the second error module includes:

[0274] The second classification submodule is used to input the second training features of the second training data into the semantic classification task corresponding to the target training task to obtain the target semantic category corresponding to the second training data, so as to obtain at least one target semantic category corresponding to the second training data respectively.

[0275] The semantic recognition submodule is used to calculate the error of the target semantic category and the semantic category ground value corresponding to at least one second training data based on the error loss function, so as to obtain the semantic recognition error.

[0276] The center loss submodule is used to calculate the error of the target semantic category and the semantic category ground truth value corresponding to at least one second training data based on the center loss function, so as to obtain the center loss error.

[0277] The apparatus in this public embodiment can be used to implement the corresponding method. For details on each step and its technical effects, please refer to the description in the relevant method, which will not be repeated here.

[0278] It should be noted that the visual retrieval model and semantic retrieval model in this embodiment are not models specific to any particular user and do not reflect the personal information of any particular user. It should also be noted that many images in the image library in this embodiment come from publicly available datasets.

[0279] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0280] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0281] According to embodiments of this disclosure, this disclosure also provides a computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the electronic device to perform the scheme provided in any of the above embodiments.

[0282] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0283] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0284] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0285] The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as image retrieval, training of a visual retrieval model, or training of a semantic retrieval model. For example, in some embodiments, the image retrieval method, the visual retrieval model training method, or the semantic retrieval model training method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the image retrieval method, the visual retrieval model training method, or the semantic retrieval model training method described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured in any other suitable manner (e.g., by means of firmware) to perform an image retrieval method, a visual retrieval model training method, or a semantic retrieval model training method.

[0286] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0287] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0288] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0289] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0290] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0291] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0292] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0293] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a visual retrieval model, comprising: performing visual clustering on a plurality of candidate images in an image library based on a visual similarity clustering algorithm to obtain at least one visual category; generating a visual retrieval model to be trained including a first encoder and a plurality of training tasks; determining at least one first training data; the first training data including first training images labeled with visual ground truth and second training images; inputting the at least one first training data into the visual retrieval model to be trained, and taking a first loss value generated by the plurality of training tasks on the visual retrieval model as a whole to satisfy a first error condition as a training target, to train visual model parameters of the visual retrieval model; wherein the first encoder of the visual retrieval model corresponding to the visual model parameters is configured to extract visual features of a to-be-retrieved image, and the visual features are configured to be spliced with semantic features obtained according to a semantic retrieval model to obtain target features of the to-be-retrieved image, so that a target image satisfying an image similarity condition with the to-be-retrieved image is searched from the image library based on the target features.

2. The method of claim 1, wherein, the plurality of training tasks including a classification task and a similarity measurement task; the generating of the visual retrieval model to be trained including the first encoder and the plurality of training tasks comprises: constructing a classification task of the visual retrieval model based on the at least one visual category; constructing a similarity measurement task of the visual retrieval model based on a similarity measurement model to obtain the plurality of training tasks corresponding to the classification task and the similarity measurement task; constructing the first encoder of the visual retrieval model based on a neural network algorithm; determining the visual retrieval model to be trained constituted by the first encoder and the plurality of training tasks.

3. The method of claim 2, wherein, the inputting of the at least one first training data into the visual retrieval model to be trained, and the taking of the first loss value generated by the plurality of training tasks on the visual retrieval model as a whole to satisfy the first error condition as the training target, to train the visual model parameters of the visual retrieval model, comprises: initializing parameters of the visual retrieval model to be trained to obtain first model parameters corresponding to the visual model parameters; inputting the at least one first training data into the first encoder of the visual retrieval model corresponding to the first model parameters in sequence to extract first training features corresponding to the at least one first training data respectively; calculating a classification error of the at least one first training data in the classification task and a similarity measurement error of the at least one first training data in the similarity measurement task based on the first training features corresponding to the at least one first training data respectively and the visual ground truth; weighting the classification error and the similarity measurement error to obtain the first loss value generated by the training; if it is determined that the first loss value satisfies the first error condition, the first model parameters are determined as the visual model parameters of the visual retrieval model. 4.The method of claim 3, further comprising: If it is determined that the first loss value does not satisfy the first error condition, the first model parameter of the visual retrieval model is updated, and the step of sequentially inputting at least one first training data into the first encoder of the visual retrieval model corresponding to the first model parameter is returned to continue to be executed.

5. The method of claim 3 or 4, wherein, The calculation of the classification error corresponding to the classification task and the similarity measurement error corresponding to the similarity measurement task of at least one first training data based on the first training feature corresponding to at least one first training data respectively and the visual true value comprises: The first training feature of the first training data is input into the classification task to obtain the target visual category corresponding to at least one first training data respectively; the target visual category is determined from at least one visual category; The classification error is calculated based on the target visual category corresponding to at least one first training data respectively and the visual category true value; The first training feature of the first training data is input into the similarity measurement task to obtain the similarity measurement result of the similarity measurement task to the first training image and the second training image in the first training data, so as to obtain the similarity measurement result corresponding to at least one training data respectively; The similarity measurement error is calculated based on the similarity measurement result corresponding to at least one first training data respectively and the similarity measurement true value.

6. A semantic retrieval model training method, comprising: Based on the semantic category mining algorithm, the semantic category of the plurality of candidate images in the image library is mined to obtain at least one semantic category; Generating a to-be-trained semantic retrieval model comprising a second encoder and a target training task; the target training task corresponds to a plurality of loss functions; Determine at least one second training data; The second training data comprises a third training image labeled with a semantic category true value; At least one second training data is input into the to-be-trained semantic retrieval model, so as to obtain a second loss value based on the overall loss calculation of the plurality of loss functions on the semantic retrieval model, and the second loss value satisfies the second error condition as the training target to train the semantic model parameter of the semantic retrieval model; Wherein, the second encoder in the semantic retrieval model corresponding to the semantic model parameter is used to extract the semantic feature of the to-be-retrieved image; the semantic feature is used to splice the target feature of the to-be-retrieved image with the visual feature obtained according to the visual retrieval model, so as to find the target image satisfying the image similarity condition with the to-be-retrieved image from the image library based on the target feature.

7. The method of claim 6, wherein, The generation of the to-be-trained semantic retrieval model comprising a second encoder and a target training task comprises: Based on at least one semantic category, the target training task of the semantic retrieval model is constructed; the target training task corresponds to a plurality of loss functions; Based on the neural network algorithm, the second encoder of the semantic retrieval model is constructed; The semantic retrieval model composed of the second encoder and the target training task is determined.

8. The method of claim 6 or 7, wherein, The second loss value obtained by performing overall loss calculation on the semantic retrieval model based on the plurality of loss functions satisfies a second error condition as a training target, and semantic model parameters of the semantic retrieval model are obtained by training. The semantic model parameters are initialized to obtain second model parameters corresponding to the semantic model parameters; At least one second training data is input into a second encoder of the semantic retrieval model corresponding to the second model parameters in sequence to extract second training features corresponding to the at least one second training data respectively; Based on the second training features corresponding to the at least one second training data respectively and the semantic category true value, semantic recognition errors and center loss errors of the at least one second training data corresponding to the target training task are calculated; The semantic recognition errors and the center loss errors are weighted to obtain a second loss value generated by training; If it is determined that the second loss value satisfies the second error condition, the second model parameters are determined as the semantic model parameters of the semantic retrieval model.

9. The method of claim 8, wherein, The target training task includes a semantic classification task based on at least one semantic category; and the calculation of the semantic recognition errors and the center loss errors of the at least one second training data corresponding to the target training task based on the second training features corresponding to the at least one second training data respectively and the semantic category true value includes: The second training features of the second training data are input into a semantic classification task corresponding to the target training task to obtain target semantic categories corresponding to the second training data, so as to obtain target semantic categories corresponding to the at least one second training data respectively; Based on an error loss function, error calculation is performed on the target semantic categories corresponding to the at least one second training data respectively and the semantic category true value to obtain the semantic recognition errors; Based on a center loss function, error calculation is performed on the target semantic categories corresponding to the at least one second training data respectively and the semantic category true value to obtain the center loss errors.

10. An image retrieval method, comprising: obtaining an image to be retrieved; extracting visual features of the image to be retrieved based on a visual retrieval model obtained by joint training of a plurality of training tasks; wherein the visual retrieval model is obtained by joint training of a plurality of training tasks based on at least one visual category obtained by visual similarity clustering; and the training of the visual retrieval model is implemented by using the method of any one of claims 1-5; extracting semantic features of the image to be retrieved based on a semantic retrieval model obtained by joint training of a plurality of loss functions; wherein the semantic retrieval model is obtained by joint training of a plurality of loss functions corresponding to a target training task based on at least one semantic category obtained by semantic category mining; and the training of the semantic retrieval model is implemented by using the method of any one of claims 6-9; splicing the visual features and the semantic features to obtain target features of the image to be retrieved; and search, from the image library, a target image that meets an image similarity condition with the image to be searched based on the target feature.

11. The method of claim 10, wherein the obtaining the image to be searched comprises: receiving an image search request sent by a user device; in response to the image search request, obtaining the image to be searched.

12. The method of claim 10 or 11, wherein, The visual search model obtained based on the joint training of the plurality of training tasks comprises: obtaining a first encoder of the visual search model obtained based on the joint training of the plurality of training tasks; inputting the image to be searched into the first encoder, and calculating the visual feature of the image to be searched by using the first encoder.

13. The method of claim 10 or 11, wherein, The semantic search model obtained based on the joint training of the plurality of loss functions comprises: obtaining a second encoder of the semantic search model obtained based on the joint training of the plurality of loss functions; inputting the image to be searched into the second encoder, and calculating the semantic feature of the image to be searched by using the second encoder.

14. The method of claim 10 or 11, wherein, The searching, from the image library, a target image that meets an image similarity condition with the image to be searched based on the target feature comprises: determining a plurality of candidate images in the image library; extracting, based on the visual search model, a plurality of candidate visual features respectively corresponding to the plurality of candidate images; extracting, based on the semantic search model, a plurality of candidate semantic features respectively corresponding to the plurality of candidate images; splicing the candidate visual feature and the candidate semantic feature of each of the plurality of candidate images to obtain a plurality of candidate features respectively corresponding to the plurality of candidate images; determining, according to the plurality of candidate features respectively corresponding to the plurality of candidate images, a target feature that meets a feature similarity condition with the target feature, so as to obtain a candidate image corresponding to the target feature as the target image.

15. The method of claim 14, wherein, The determining, according to the plurality of candidate features respectively corresponding to the plurality of candidate images, a target feature that meets a feature similarity condition with the target feature comprises: performing spatial clustering on the plurality of candidate features respectively corresponding to the plurality of candidate images to obtain a spatial tree formed by a plurality of spatial nodes, image categories corresponding to the plurality of spatial nodes, and spatial features; the image category of a parent spatial node is a parent category of the image categories of its child spatial nodes; each of the image categories corresponding to the plurality of spatial nodes comprises at least one candidate feature respectively corresponding to a candidate image; determining, based on the spatial features respectively corresponding to the plurality of spatial nodes in the spatial tree, a target spatial node that meets a spatial similarity condition with the target feature; determining, from the candidate features respectively corresponding to at least one candidate image of the target spatial node, a target feature that meets a feature similarity condition with the target feature.

16. The method of claim 15, wherein, The determining, based on the spatial features respectively corresponding to the plurality of spatial nodes in the spatial tree, a target spatial node that meets a spatial similarity condition with the target feature comprises: calculating, from the root node of the spatial tree, the feature distance between the spatial feature of each spatial node and the target feature in sequence; if the feature distance of any spatial node is determined to be less than a distance threshold, the spatial node is determined to be a target spatial node that meets a spatial similarity condition with the target feature.

17. A device for training a visual retrieval model, comprising: a category clustering unit configured to perform visual clustering on a plurality of candidate images in an image library based on a visual similarity clustering algorithm to obtain at least one visual category; a first generation unit configured to generate a visual retrieval model to be trained, including a first encoder and a plurality of training tasks; a first determination unit configured to determine at least one first training data; the first training data including first training images labeled with visual ground truth and second training images; the visual ground truth including visual category ground truth determined based on the at least one visual category and a similarity measure ground truth; a first training unit configured to input the at least one first training data into the visual retrieval model to be trained, and train the visual retrieval model to obtain visual model parameters of the visual retrieval model, with a first loss value generated by the plurality of training tasks on the visual retrieval model as a training target, the first loss value satisfying a first error condition; wherein the first encoder of the visual retrieval model corresponding to the visual model parameters is configured to extract visual features of a to-be-retrieved image, and the visual features are configured to be spliced with semantic features obtained according to a semantic retrieval model to obtain target features of the to-be-retrieved image, so that a target image satisfying an image similarity condition with the to-be-retrieved image is searched from the image library based on the target features.

18. The apparatus of claim 17, wherein, the plurality of training tasks including a classification task and a similarity measure task; and the first generation unit including: a first construction module configured to construct the classification task of the visual retrieval model based on the at least one visual category; a second construction module configured to construct a similarity measure task of the visual retrieval model based on a similarity measure model to obtain the plurality of training tasks corresponding to the classification task and the similarity measure task; a first encoding module configured to construct the first encoder of the visual retrieval model based on a neural network algorithm; a first determination module configured to determine the visual retrieval model to be trained formed by the first encoder and the plurality of training tasks.

19. The apparatus of claim 18, wherein, the first training unit including: a first initialization module configured to initialize parameters of the visual retrieval model to be trained to obtain first model parameters corresponding to the visual model parameters; a first training module configured to input the at least one first training data into the first encoder of the visual retrieval model corresponding to the first model parameters in sequence to extract first training features corresponding to the at least one first training data respectively; a first error module configured to calculate a classification error of the at least one first training data in the classification task and a similarity measure error of the at least one first training data in the similarity measure task based on the first training features corresponding to the at least one first training data respectively and the visual ground truth; a first weighting module configured to weight the classification error and the similarity measure error to obtain a first loss value generated by training; a first determination module configured to determine the first model parameters as the visual model parameters of the visual retrieval model if the first loss value satisfies the first error condition.

20. The apparatus of claim 19, wherein, the first training unit further including The second determining module is configured to update the first model parameter of the visual retrieval model if it is determined that the first loss value does not satisfy the first error condition, and return to the step of inputting the at least one first training data into the first encoder of the visual retrieval model corresponding to the first model parameter in sequence for continuous execution.

21. The apparatus of claim 19 or 20, wherein, The first error module comprises: The first training submodule is configured to input the first training feature of the first training data into the classification task to obtain a classification result corresponding to each of the at least one first training data; the classification result is a target visual category determined from the at least one visual category; The classification error submodule is configured to calculate the classification error based on the target visual category corresponding to each of the at least one first training data and the visual category true value; The similarity measurement submodule is configured to input the first training feature of the first training data into the similarity measurement task to obtain a similarity measurement result of the similarity measurement task on the first training image and the second training image in the first training data, so as to obtain a similarity measurement result corresponding to each of the at least one training data; The error calculation submodule is configured to calculate the similarity measurement error based on the similarity measurement result corresponding to each of the at least one first training data and the similarity measurement true value.

22. A training device of a semantic retrieval model, comprising: A category mining unit configured to perform semantic category mining on a plurality of candidate images in an image library based on a semantic category mining algorithm to obtain at least one semantic category; A second generation unit configured to generate a semantic retrieval model to be trained comprising a second encoder and a target training task; the target training task corresponds to a plurality of loss functions; A second determination unit configured to determine at least one second training data; The second training data comprises a third training image labeled with a semantic category true value; A second training unit configured to input the at least one second training data into the semantic retrieval model to be trained, so as to train semantic model parameters of the semantic retrieval model based on a second loss value obtained by performing overall loss calculation on the semantic retrieval model based on the plurality of loss functions, with a second error condition being a training target; The semantic model parameters correspond to a second encoder in the semantic retrieval model, which is configured to extract semantic features of a to-be-retrieved image; the semantic features are configured to be spliced with visual features obtained according to a visual retrieval model into target features of the to-be-retrieved image, so as to search for a target image satisfying an image similarity condition with the to-be-retrieved image from an image library based on the target features.

23. The apparatus of claim 22, wherein, The second generation unit comprises: Based on the at least one semantic category, a target training task of a semantic retrieval model is constructed; the target training task corresponds to a plurality of loss functions; Based on a neural network algorithm, a second encoder of the semantic retrieval model is constructed; The semantic retrieval model composed of the second encoder and the target training task is determined.

24. The apparatus of claim 22 or 23, wherein, The second training unit comprises: A second initialization module configured to initialize parameters of the semantic retrieval model to be trained to obtain second model parameters corresponding to the semantic model parameters; The second training module is configured to input the at least one second training data into a second encoder of a semantic retrieval model corresponding to the second model parameter in sequence to extract second training features corresponding to the at least one second training data respectively; The second error module is configured to calculate semantic recognition errors and center loss errors of the at least one second training data in the target training task based on the second training features corresponding to the at least one second training data respectively and the semantic category true value; The second weighting module is configured to weight the semantic recognition errors and the center loss errors to obtain a second loss value generated by training; The second determination module is configured to determine that the second model parameter is a semantic model parameter of the semantic retrieval model if it is determined that the second loss value satisfies a second error condition.

25. The apparatus of claim 24, wherein, The target training task includes a semantic classification task based on at least one semantic category; The second error module includes: The second classification submodule is configured to input the second training features of the second training data into a semantic classification task corresponding to the target training task to obtain target semantic categories corresponding to the second training data, so as to obtain target semantic categories corresponding to the at least one second training data respectively; The semantic recognition submodule is configured to perform error calculation on the target semantic categories corresponding to the at least one second training data respectively and the semantic category true value based on an error loss function to obtain the semantic recognition errors; The center loss submodule is configured to perform error calculation on the target semantic categories corresponding to the at least one second training data respectively and the semantic category true value based on a center loss function to obtain the center loss errors.

26. An image retrieval apparatus, comprising: An image acquisition unit configured to acquire a to-be-retrieved image; A first extraction unit configured to extract visual features of the to-be-retrieved image based on a visual retrieval model obtained through joint training of multiple training tasks, wherein the visual retrieval model is obtained through joint training of multiple training tasks based on at least one visual category obtained through visual similarity clustering, and training of the visual retrieval model is implemented by using the method according to any one of claims 1-5; A second extraction unit configured to extract semantic features of the to-be-retrieved image based on a semantic retrieval model obtained through joint training of multiple loss functions, wherein the semantic retrieval model is obtained through joint training of multiple loss functions corresponding to a target training task based on at least one semantic category obtained through semantic category mining, and training of the semantic retrieval model is implemented by using the method according to any one of claims 6-9; A first splicing unit configured to splice the visual features and the semantic features to obtain target features of the to-be-retrieved image; A target searching unit configured to search, based on the target features, a target image that satisfies an image similarity condition with the to-be-retrieved image from an image library.

27. The apparatus of claim 26, wherein, The image acquisition unit includes: A request receiving module configured to receive an image retrieval request sent by a user equipment; An image acquisition module configured to acquire the to-be-retrieved image in response to the image retrieval request.

28. The apparatus of claim 26 or 27, wherein, The first extraction unit comprises: The first acquisition module is configured to acquire a first encoder of the visual retrieval model obtained through joint training of a plurality of training tasks; The first input module is configured to input the image to be retrieved into the first encoder, and calculate the visual feature of the image to be retrieved by using the first encoder.

29. The apparatus of claim 26 or 27, wherein, The second extraction unit comprises: The second acquisition module is configured to acquire a second encoder of the semantic retrieval model obtained through joint training of a plurality of loss functions; The second input module is configured to input the image to be retrieved into the second encoder, and calculate the semantic feature of the image to be retrieved by using the second encoder.

30. The apparatus of claim 26 or 27, wherein, The target searching unit comprises: The candidate determination module is configured to determine a plurality of candidate images in the image library; The first extraction module is configured to extract a candidate visual feature corresponding to each of the candidate images based on the visual retrieval model; The second extraction module is configured to extract a candidate semantic feature corresponding to each of the candidate images based on the semantic retrieval model; The first splicing module is configured to splice the candidate visual feature and the candidate semantic feature of each of the candidate images to obtain a candidate feature corresponding to each of the candidate images; The feature matching module is configured to search for a target feature satisfying a feature similarity condition with the target feature according to the candidate features corresponding to the plurality of candidate images, so as to obtain the candidate image corresponding to the target feature as the target image.

31. The apparatus of claim 30, wherein, The feature matching module comprises: The spatial clustering submodule is configured to perform spatial clustering on the candidate features corresponding to the plurality of candidate images to obtain a spatial tree formed by a plurality of spatial nodes, image categories corresponding to the spatial nodes, and spatial features; the image category of a parent spatial node is a parent category of the image categories of its child spatial nodes; and each image category corresponding to a spatial node comprises at least one candidate feature corresponding to a candidate image; The spatial matching submodule is configured to determine a target spatial node satisfying a spatial similarity condition with the target feature based on the spatial features corresponding to the plurality of spatial nodes in the spatial tree; The feature determination submodule is configured to determine a target feature satisfying a feature similarity condition with the target feature from the candidate features corresponding to at least one candidate image of the target spatial node.

32. The apparatus of claim 31, wherein, The spatial matching submodule is specifically configured to: Start from a root node of the spatial tree, and sequentially calculate a feature distance between the spatial feature of each spatial node and the target feature; if it is determined that the feature distance of any spatial node is less than a distance threshold, the spatial node is determined as a target spatial node satisfying the spatial similarity condition with the target feature. 33.An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5, 6-9, or 10-16.

34. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are for causing the computer to perform the method according to any one of claims 1-5, 6-9, or 10-16.

35. A computer program product comprising computer instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1-5, 6-9, or 10-16.

Images