Model training method, search method, device, electronic equipment and storage medium

Abstract

The present disclosure provides a model training method, a search method, an apparatus, an electronic device and a storage medium, relates to the technical field of artificial intelligence, and in particular to the fields of deep learning, computer vision, intelligent search and the like. The specific implementation scheme is as follows: input a sample picture into a first model to be trained; global feature extraction, local region identification and reconstruction of the sample picture are performed on the sample picture in the first model to obtain a target loss function; and the first model is trained according to the back propagation of the target loss function to obtain a second model after training. By using the present disclosure, the search speed is fast and the accuracy is high.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of deep learning, computer vision, and intelligent search. Background Technology

[0002] With the development of technology, artificial intelligence can be used to accelerate hardware processing. It is applicable to a wide variety of application scenarios, such as hardware design involving image processing, text processing, and video processing, in order to improve the processing speed and accuracy of the hardware.

[0003] For example, in the field of search, current image search methods can no longer meet people's needs. There is a need for a model that is both fast and accurate in search processing to enable faster and more precise image searches. However, no effective solution exists in this area. Summary of the Invention

[0004] This disclosure provides a model training method, a search method, an apparatus, an electronic device, and a storage medium.

[0005] According to one aspect of this disclosure, a model training method is provided, comprising:

[0006] Input the sample images into the first model to be trained;

[0007] In the first model, global feature extraction, local region identification, and reconstruction of the sample image are performed to obtain the target loss function;

[0008] The first model is trained by backpropagation of the target loss function to obtain the trained second model.

[0009] According to another aspect of this disclosure, a search method is provided, comprising:

[0010] The image to be matched is input into the second model, which is a model obtained by applying the above model training method.

[0011] Based on the second model, local features are extracted from the image to be matched;

[0012] Image search is performed based on the local features to obtain similar images that match the image to be matched in terms of the local features.

[0013] According to another aspect of this disclosure, a model training apparatus is provided, comprising:

[0014] The first input unit is used to input sample images into the first model to be trained.

[0015] The first processing unit is used to perform global feature extraction, local region recognition, and reconstruction of the sample image in the first model to obtain the target loss function.

[0016] The second processing unit is used to train the first model based on backpropagation of the target loss function to obtain the trained second model.

[0017] According to another aspect of this disclosure, a search apparatus is provided, comprising:

[0018] The second input unit is used to input the image to be matched into the second model, which is a model obtained by applying the above model training method.

[0019] An extraction unit is used to extract local features from the image to be matched based on the second model;

[0020] The search unit is used to perform image search based on the local features to obtain similar images that match the image to be matched in terms of the local features.

[0021] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0022] At least one processor; and

[0023] The memory is communicatively connected to the at least one processor; wherein,

[0024] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method provided in any embodiment of this disclosure.

[0025] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods provided in any embodiment of this disclosure.

[0026] According to another aspect of this disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, implement the methods provided in any embodiment of this disclosure.

[0027] Using this disclosure, a sample image can be input into a first model to be trained. In the first model, global feature extraction, local region identification, and sample image reconstruction are performed on the sample image to obtain a target loss function. The first model is then trained based on backpropagation of the target loss function to obtain a trained second model. The second model is used in the search process. Since the second model is trained using the target loss function obtained through global feature extraction, local region identification, and sample image reconstruction, global features can be obtained through global feature extraction, local regions can be obtained through local region identification, and more accurate sample images can be obtained through sample image reconstruction. Therefore, by combining global feature extraction, local region identification, and the reconstruction of the sample image, the second model trained with the target loss function has a faster search speed and higher accuracy.

[0028] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0029] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0030] Figure 1 This is a schematic flowchart of a model training method according to an embodiment of the present disclosure;

[0031] Figure 2 This is a schematic diagram of an image to be matched according to an embodiment of the present disclosure;

[0032] Figure 3 This is a schematic diagram of another image to be matched according to an embodiment of this disclosure;

[0033] Figure 4 This is a flowchart illustrating the search method according to an embodiment of the present disclosure;

[0034] Figure 5 This is a schematic diagram of model training for local feature extraction in an application example according to an embodiment of the present disclosure;

[0035] Figure 6 This is a schematic diagram illustrating similar image search in an application example according to an embodiment of this disclosure;

[0036] Figure 7 This is a schematic diagram of the composition structure of a model training device according to an embodiment of the present disclosure;

[0037] Figure 8 This is a schematic diagram of the composition of a search device according to an embodiment of the present disclosure;

[0038] Figure 9This is a block diagram of an electronic device used to implement the model training or search method of the embodiments of this disclosure. Detailed Implementation

[0039] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0040] In this document, the term "and / or" merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document indicates any combination of at least two of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this document refer to and distinguish between multiple similar technical terms, not to restrict the order or to limit there to only two. For example, "first feature" and "second feature" refer to two categories / two features; the first feature can be one or more, and the second feature can also be one or more.

[0041] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0042] Search processing includes image search, video search, and more. With advancements in computer vision technology, image search has become significantly more accurate than earlier methods, leading to its wider applications. Examples include searching for products, plants, and celebrities by taking a photo. Whether searching for products, plants, or celebrities, the process essentially involves finding similar images that match the current image. Higher search accuracy results in more relevant and relevant images being provided to the user.

[0043] According to embodiments of this disclosure, a model training method is provided. Figure 1This is a flowchart illustrating a model training method according to an embodiment of the present disclosure. This method can be applied to a model training device, for example, deployed in a single-machine, multi-machine, or cluster system as a terminal, server, or other processing device. It can perform processing such as searching for images, text, and videos in various application scenarios. The terminal can be a user equipment (UE), mobile device, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, it includes:

[0044] S101. Input the sample images into the first model to be trained.

[0045] S102. In this first model, global feature extraction, local region recognition, and reconstruction of the sample image are performed to obtain the target loss function.

[0046] S103. Train the first model using backpropagation based on the target loss function to obtain the trained second model.

[0047] In one example of S101-S103, a sample image is input into the first model to be trained. In the first model, multiple branches are used for the sample image. Global feature extraction, local region recognition, and reconstruction of the sample image are performed on each branch. Finally, the target loss function is obtained. The first model is trained by backpropagation based on the target loss function to obtain the trained second model (such as a search model for extracting local features of an image in image search processing).

[0048] Using the embodiments of this disclosure, a sample image can be input into a first model to be trained. In this first model, global feature extraction, local region identification, and sample image reconstruction are performed on the sample image to obtain a target loss function. The first model is then trained using backpropagation based on this target loss function to obtain a trained second model. This second model is then used for search processing. Since the second model is trained using a target loss function obtained through global feature extraction, local region identification, and sample image reconstruction, where global feature extraction yields global features, local region identification yields local regions, and sample image reconstruction yields more accurate sample images, the second model trained using the target loss function obtained by combining global feature extraction, local region identification, and sample image reconstruction has a fast search speed and high accuracy.

[0049] In one embodiment, the first model performs global feature extraction, local region identification, and reconstruction of the sample image to obtain a target loss function. This includes: obtaining at least one feature map from the sample image in the first model; performing global feature extraction, local region identification, and reconstruction of the sample image based on the at least one feature map to obtain the target loss function. By using this embodiment, at least one feature map is obtained from the sample image to extract global features, identify local regions, and reconstruct the sample image. Based on more accurate global and local features, combining these global and local features with the feature reconstruction obtained from the reconstructed sample image yields more comprehensive features describing the image content and clarifies key local semantics, thereby improving the accuracy of finding similar images with higher precision to the image to be matched.

[0050] In one example, Figure 2 This is a schematic diagram of an image to be matched according to an embodiment of the present disclosure, such as... Figure 2 As shown, the liquor packaging box 201 has information about the liquor "Fenggu Chunniang" 202 on it. The packaging box is yellow. The user wants to search for similar images that match the liquor "Fenggu Chunniang".

[0051] Because the model can only extract single global features, its ability to represent the local semantic information of the main subject in the image is poor. It cannot represent information such as "Fenggu Pure Brew" on the packaging box. Therefore, the recall results are mostly images of yellow boxes or cartons, and most of them are not related to alcohol, resulting in low search precision. To address this, this implementation method can extract not only global features but also local features, as well as features reconstructed from the sample images. This provides a more comprehensive description of the image content, clarifies key local semantics, and thus improves the accuracy of finding similar images that match the image to be matched.

[0052] In another example, Figure 3 This is a schematic diagram of another image to be matched according to an embodiment of this disclosure, such as... Figure 3 The image shown is of a long-haired anime girl holding a violin against a blue background. The user wants to search for similar images that match the art style of "anime girl with a blue background holding a violin".

[0053] Because the model can only extract single global features, its ability to represent the local semantic information of the subject in the image is poor. It cannot represent local information such as the "qin" (a type of stringed instrument) in the anime image. Therefore, the recall results are mostly: an anime girl with a blue background, no qin, resulting in low search precision. To address this, this implementation method can extract not only global features but also local features, as well as features reconstructed from the sample image. This provides a more comprehensive description of the image content, clearly defining key local semantics, thereby improving the accuracy of finding similar images with higher precision.

[0054] It should be noted that the pooling layer used in the global branch for extracting global features is a learnable pooling layer that falls between average pooling and max pooling. Based on this pooling layer, more content related to global features can be learned to improve the semantic information of global features. The attention branch for extracting local features does not extract all local feature points, but filters out irrelevant points in local features and retains similar points to improve the semantic discriminativeness of local features. The reconstruction branch for reconstructing sample images takes into account that different sample images have different resolutions, and the semantic information of the features contained in sample images with different resolutions is different. In order to avoid feature loss, the sample images are reconstructed to improve feature reconstruction and obtain features that more comprehensively describe the content of the image.

[0055] In one embodiment, the global feature extraction, local region identification, and sample image reconstruction based on the at least one feature map to obtain the target loss function includes: extracting global features of the first feature map in the first branch of the first model, obtaining a first loss function based on the global features, identifying a first local region in the second feature map that is different from other regions in the second branch of the first model, obtaining a second loss function by combining the third feature map output by the third branch of the first model, obtaining a third loss function by combining the second feature map with the third feature map output by the third branch of the first model, and obtaining the target loss function based on the first loss function, the second loss function, and the third loss function. In this implementation, global features are extracted, local regions are identified, and sample images are reconstructed based on at least one feature map and three branches (e.g., the first branch is the global branch, the second branch is the attention branch, and the third branch is the reconstruction branch). This results in more accurate global and local features. By combining these features, a more comprehensive description of the image content can be obtained. Furthermore, the model is trained using the target loss function obtained from the loss functions of the three branches (e.g., the first loss function for the first branch, the second loss function for the second branch, and the third loss function for the third branch). The resulting second model can extract more accurate local features, improving search accuracy.

[0056] In one embodiment, extracting global features from a first feature map in a first branch of the first model and obtaining a first loss function based on the global features includes: extracting global features from the first feature map in the first branch of the first model, performing loss calculation based on the global features, and obtaining the first loss function. Using this embodiment, global features can be extracted based on at least one feature map and a first branch (such as a global branch), and loss calculation can be performed based on the global features. The obtained first loss function can then be used for the target loss function calculation in model training.

[0057] In one embodiment, a first local region in the second feature map that differs from other regions is identified in the second branch of the first model. A second loss function is obtained by combining this with the third feature map output by the third branch of the first model. This includes: identifying the first local region in the second feature map that differs from other regions using an attention mechanism in the second branch of the first model, obtaining the weights corresponding to each similar point in the first local region; reconstructing features from the second feature map in the third branch of the first model, outputting the third feature map; obtaining local features based on the weights corresponding to each similar point and the third feature map; and performing loss calculations based on the local features to obtain the second loss function. Using this embodiment, a first local region in the second feature map that differs from other regions can be identified based on at least one feature map and a second branch (such as an attention branch). Local features can be obtained from the third feature map obtained by reconstructing features from the second feature map using the weights corresponding to each similar point in the first local region and the third feature map obtained by the third branch. Loss calculations can be performed based on these local features, and the resulting second loss function can be used for the target loss function calculation in model training.

[0058] In one embodiment, a third loss function is obtained by combining the second feature map with the third feature map output by the third branch in the first model. This includes: performing feature reconstruction on the second feature map in the third branch of the first model to output the third feature map; performing loss calculation based on the second feature map and the third feature map to obtain the third loss function. Using this embodiment, feature reconstruction can be performed based on at least one feature map and a third branch (such as a reconstruction branch). After obtaining the third feature map, loss calculation is performed, and the obtained third loss function is used for the target loss function calculation in model training.

[0059] According to embodiments of this disclosure, a search method is provided. Figure 4 This is a flowchart illustrating a search method according to an embodiment of the present disclosure. This method can be applied to a search device, for example, deployed on a terminal, server, or other processing device in a single-machine, multi-machine, or cluster system. It can perform searches and other processing for various application scenarios, including images, text, and videos. The terminal can be a user equipment (UE), mobile device, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 4 As shown, it includes:

[0060] S401. Input the image to be matched into the second model, which is a model obtained by applying the above model training method.

[0061] S402. Based on the second model, extract the local features from the image to be matched.

[0062] S403. Perform an image search based on the local feature to obtain similar images that match the image to be matched in terms of the local feature.

[0063] In one example of S401-S403, the image to be matched is input into a second model (such as a search model used for image search processing to extract local features of an image). This second model can be obtained by applying the model training method described above. Based on this second model, local features in the image to be matched can be extracted (such as local feature points in an image that describe local information; or, for example, pairs of local feature points, or matching point pairs, existing in two similar images). Image search is then performed based on these local features to obtain similar images that match the image to be matched in terms of those local features.

[0064] Using the embodiments of this disclosure, the image to be matched is input into a second model (the second model is a model obtained by applying the above-described model training method). Local features in the image to be matched can be extracted according to the second model so that image search can be performed based on the local features to obtain similar images that match the image to be matched in terms of the local features. In terms of search processing, the second model is used, and the search speed is fast and the accuracy is high.

[0065] In one embodiment, the method further includes: performing a matching operation based on the local features to obtain a mapping matrix, and performing feature mapping based on the mapping matrix to obtain point features among the local features. By employing this embodiment and using a matching algorithm, the local features can be further filtered to obtain more accurate local point features. Searching based on these local point features can further improve search accuracy and speed.

[0066] In one embodiment, the local feature includes: feature points matched by image pairs with similar points in the images to be matched. In this embodiment, the local feature can be a pair of local feature points, or a matching point pair, in two similar images that share similar local information. This matching point pair has more commonalities in its local features, and using this matching point pair allows for further filtering of the local feature.

[0067] In one embodiment, performing a matching operation based on the local features to obtain a mapping matrix includes: forming matching point pairs based on the feature points matched by image pairs with similar points in the image to be matched; and performing a matching operation based on the matching point pairs to obtain the mapping matrix. In this embodiment, the matching point pairs are used for the matching operation because they share more commonalities in local features. Using these matching point pairs for the matching operation allows for more accurate point features to be obtained based on the obtained mapping matrix when performing feature mapping.

[0068] In one embodiment, the method further includes: performing Product Quantization (PQ) operations on the point features in the local features to obtain target point features, wherein the storage size of the target point features is less than the storage size of the point features in the local features. By employing this embodiment, the storage size of the point features can be reduced through PQ operations, making the storage size of the target point features smaller than the storage size of the initial point features (i.e., the point features in the local features). This reduction in storage size correspondingly improves the search speed.

[0069] In one embodiment, a product-PQ quantization operation is performed on the point features in the local features to obtain the target point features. This includes: acquiring the point features in the local features described as floating-point numbers (float); performing a PQ operation on the point features in the local features described as floating-point numbers to convert them into point features in the local features described as integers (int); and using the point features in the local features described as int as the target point features. This embodiment achieves the format conversion of point features through PQ quantization, converting the point features from a floating-point description to an int description, thereby reducing storage requirements.

[0070] The model training method and search method provided in the above-described embodiments of this disclosure are illustrated below.

[0071] Local features of an image are used to describe its invariance and discriminability. Invariance refers to its robustness to color and scale changes; discriminability refers to its ability to strongly represent the local features of the image. Some algorithms for extracting local features include: Scale-invariant feature transform (SIFT), Histogram of Oriented Gradients (HOG), and Deep Local Feature (DELF) based on neural networks.

[0072] The SIFT algorithm is a visual algorithm used to detect and describe local features in images. It finds extreme points in spatial scales and extracts their position, scale, and rotation invariants. These local features, as feature descriptors, are highly robust, remaining invariant to rotation, scale changes, and brightness variations, as well as to viewpoint changes, affine transformations, and noise. SIFT's advantages include rich information content, high stability, strong scalability, and ease of combination with other feature vectors. Its disadvantages include low real-time performance and poor robustness to images with smooth edges.

[0073] The HOG algorithm constructs features by calculating and statistically analyzing the gradient orientation histograms of local regions in an image. Because HOG operates on local grid cells of the image, it maintains good invariance to geometric and optical deformations (which only occur in a larger spatial domain). Under conditions of coarse spatial sampling, fine orientation sampling, and strong local optical normalization, taking pedestrian detection as an example, as long as the pedestrian maintains a generally upright posture, minor limb movements can be tolerated and ignored without affecting the detection results. Therefore, HOG features are particularly suitable for human detection in images.

[0074] Regarding the DELF algorithm, it is an attention-based local feature representation used for large-scale image retrieval. These local features are extracted from a trained convolutional neural network. By introducing an attention mechanism to determine which key points or regions in an image are more important, an image pyramid is constructed, and a fully convolutional network is applied to each layer of the image pyramid, which can solve the problem of size variation. The advantage of this algorithm is that it can dynamically extract point features from the image and adaptively extract the location of key points for different types of images. The disadvantage is that the feature dimension is relatively large.

[0075] In this application example, based on the DELF algorithm described above, an attention mechanism is also introduced, and a search model for extracting local features is designed. The local region where these local features are located is a first local region that the search model learns from sample images and can distinguish from other surrounding areas. This first local region has high discriminability, and the local features in this first local region are robust to various categories, including the following:

[0076] I. Training Data Construction for the Search Model: The sample images can be classified into 50,000 categories, forming a training dataset of 10 million images. This training dataset primarily contains data with strong local feature learning capabilities to ensure that useful local feature points can be extracted from each image. The amount of data in each category is between 30 and 200 to prevent feature imbalance caused by class imbalance, thus ensuring more accurate learning of local features. Each category can also include data on the same subject in different colors (e.g., if the subject is a car, it can be black, blue, red, champagne, etc.) to ensure the model's resilience to color variations.

[0077] II. Training the search model:

[0078] Figure 5 This is a schematic diagram of model training for local feature extraction in an application example according to an embodiment of this disclosure, such as... Figure 5 As shown, it includes the following three branches:

[0079] 1) First branch (i.e. global branch): Extract global features from the first feature map (i.e. the feature map of the penultimate layer in the backbone network) through the global branch, perform additive angular margin (ArcFace) operation based on the global features, and use the resulting additive angular margin loss function (ArcFace loss) as the first loss function.

[0080] Specifically, the backbone network of this global branch can use ResNet50, and the last part of this global branch is a pooling layer (such as GeM-Pooling). The loss operation can use ArcFace loss (using ArcFace loss can improve the discriminability between categories and strengthen intra-class compactness and inter-class differences, which helps to improve the visual resolution of features). Among them, GeM-Pooling is a learnable pooling layer between average pooling and max pooling. The weights between pixels can be dynamically adjusted instead of fixed values. That is, when the adjustment parameter p=1, GeM-Pooling degenerates into average pooling. When the adjustment parameter p is infinite, GeM-Pooling is equivalent to max pooling. Using this global branch to extract global features, more content related to global features can be learned based on this pooling layer to improve the semantic information of global features, enhance robustness to images of different resolutions, and improve the representation ability of features.

[0081] 2) Second branch (i.e. attention branch): This attention branch, which introduces the attention mechanism, can learn important location regions in the feature map (location regions that can better describe local features). The second branch identifies the first local region in the second feature map (i.e. the second-to-last layer feature map in the backbone network) that is different from other regions. After combining the third feature map output by the third branch (i.e. the feature map obtained by feature reconstruction based on the second feature map) to obtain local features, cross-entropy operation is performed based on the local features, and the resulting cross-entropy loss function is used as the second loss function.

[0082] Specifically, regarding the attention mechanism, the reason people can quickly determine the category of things when observing them is because the brain can focus attention on the most recognizable area of ​​the thing to make a judgment, thus generating the attention mechanism. The penultimate layer feature map in the aforementioned backbone network is chosen as the second feature map because it possesses more semantic information that facilitates learning local features. The final weights corresponding to each similar point in the second feature map (this second feature map is 1024-dimensional, meaning the weights corresponding to each similar point are extracted from a 1024-dimensional feature map) are then used to calculate the local features. These weights are then combined with the third feature map output from the third branch (this third feature map is 64-dimensional, meaning it's a 64-dimensional feature map obtained by reducing the 1024-dimensional feature map to extract local features more quickly). During feature extraction, not all local feature points are extracted; instead, irrelevant points are filtered out, and similar points are retained to improve the semantic discriminative power of the local features, making them more semantically distinctive. Cross Entropy operation is performed based on this local feature to obtain the Cross EntropyLoss.

[0083] 3) Third branch (i.e. reconstruction branch): The second feature map (i.e. the penultimate layer feature map in the backbone network) is reconstructed by the reconstruction branch to obtain the third feature map (i.e. the feature map obtained by reconstructing the second feature map). The mean squared error (MSE) is calculated based on the second feature map and the third feature map, and the resulting mean squared error loss function (MSE Loss) is used as the third loss function.

[0084] Specifically, a fully convolutional network can be used to reconstruct the second feature map, enabling the learned local features (such as local point features) to accurately represent the key information of the original image. This can be achieved by downsampling the second feature map to obtain a third feature map (the second feature map is reduced to 64 dimensions), and then upsampling this third feature map to the original image size. The reconstruction branch for reconstructing sample images considers that different sample images have different resolutions, and the semantic information of features contained in sample images at different resolutions differs. To avoid feature loss, reconstructing sample images to improve feature reconstruction yields more comprehensive features describing the image content, thus avoiding the loss of key information in the feature map used to identify local features. Based on this, more accurate local features can be obtained. The reconstructed third feature map and the original second feature map are used to perform MSE calculations to obtain the MSE Loss.

[0085] During the training process of the above model, the cosine loss will decrease after a period of training. At this point, calculating the ArcFace loss of the global branch can update the weights of the backbone network's res50, while the remaining two branches only update their respective weights. Considering that local features are best combined with global features, the weights for calculating the ArcFace loss of the global branch can be set to play a dominant role. Gradient pruning can also be performed on the reconstruction branch during model training to accelerate the model's training.

[0086] The search model trained using the above model extracts local features in an end-to-end manner. Because the search model has low design complexity and can accurately extract local features without consuming a lot of online resources, it not only has high search accuracy but also fast search speed in image search processing.

[0087] To improve search accuracy and speed, this application example can also employ the following weighted streaming data mining (Lossy Weight, LW) matching algorithm and PQ operation.

[0088] 1) For the LW matching algorithm, the local features (such as 64-dimensional features) obtained using the search model described above can be post-processed. The training data used can be the same as the sample images used to train the search model, i.e., a training dataset consisting of 10 million images with 50,000 categories. Alternatively, a portion of the training dataset can be used to improve matching speed. 30,000-40,000 image pairs are randomly selected from the training dataset to ensure that the similarity points between the two images in a pair can be matched, thus improving the matching of similar images. These image pairs can be input into the search model to obtain local feature point pairs (or matching point pairs). For example, feature points that can match each other in the image pairs can be filtered out to obtain approximately 100,000 matching point pairs. Then, LW matching operations are performed on these matching point pairs to obtain a mapping matrix. For example, these approximately 100,000 matching point pairs can be used to perform the LW matching operation to obtain the m-p mapping matrix. Using the m and p mapping matrices for feature mapping can yield more accurate point features in the local features (without dimensionality reduction, still 64-dimensional features).

[0089] 2) Regarding PQ operations, once image search products are launched and put into use, the amount of image data is in the tens of billions, and the extracted point features are in the hundreds of billions. Such a large number of features require a lot of storage resources. Therefore, it is necessary to convert the format of local features (such as local point features) obtained based on the above search model and further using the LW matching algorithm to reduce the storage volume. For example, converting local point features described by float to local point features described by int will reduce the storage volume to one-quarter of the original storage.

[0090] The training data used in the PQ operation can come from a search database. 1000 images are randomly selected from this database. The aforementioned search model and the LW matching algorithm are used to extract local point features described by float. Then, a symmetric distance algorithm is used to convert the float format to int format, and this conversion is written into the PQ operation codebook. This codebook stores the mapping relationship between the local point features described by float and those described by int. This codebook is then used to convert the local point features described by float to those described by int.

[0091] In one example of a search scenario Figure 6 This is a schematic diagram of similar image search in an application example according to an embodiment of this disclosure, such as... Figure 6 As shown, it includes:

[0092] S601. Input the image to be matched;

[0093] S602. Extract recall features (such as global features) and ranking features (such as local features) of the images to be matched through the search model.

[0094] S603, Search the top 400 search results;

[0095] S604. Reorder the top 400 search results;

[0096] S605, Output target search results: 400 similar images.

[0097] It should be noted that, to improve search speed, before inputting the image to be matched into the search model trained by the aforementioned model, the image can also be input into a model used to extract recall features (such as global features). The first round of image search is performed based on the obtained global features. Then, the image is input into the search model trained by the aforementioned model, which extracts ranking features (such as local features). A second round of image search is performed based on these local features to obtain the target search result. Alternatively, the search model can extract ranking features (such as local features). If the local feature is the number of matching points between two images, the score of the number of matching points is fused with the recall score obtained from the aforementioned recall features to obtain an evaluation score. This evaluation score is then used to perform a second round of re-ranking of the first round of image search to obtain the target search result. The score of the number of matching points can be obtained by extracting local features to obtain the local point features of each image, using matrix operations to obtain the distance between any two local point features of the two images, and setting a threshold. If the distance is less than the threshold, the actual score is the score of the number of matching points; if it is greater than the threshold, the score of the number of matching points is set to the threshold value.

[0098] According to embodiments of this disclosure, a model training apparatus is provided. Figure 7 This is a schematic diagram of the composition structure of a model training device according to an embodiment of the present disclosure, as shown below. Figure 7 As shown, the model training device 700 includes: a first input unit 701, used to input sample images into a first model to be trained; a first processing unit 702, used to perform global feature extraction, local region recognition, and reconstruction of the sample images in the first model to obtain a target loss function; and a second processing unit 703, used to train the first model according to the backpropagation of the target loss function to obtain a trained second model.

[0099] In one embodiment, the first processing unit is configured to obtain at least one feature map in the first model based on the sample image; and to perform global feature extraction, local region recognition, and reconstruction of the sample image based on the at least one feature map to obtain the target loss function.

[0100] In one embodiment, the first processing unit is configured to extract global features of a first feature map in a first branch of the first model, and obtain a first loss function based on the global features; identify a first local region in a second feature map that is different from other regions in a second branch of the first model, and obtain a second loss function by combining it with a third feature map output by a third branch of the first model; obtain a third loss function by combining the second feature map with the third feature map output by the third branch of the first model; and obtain the target loss function based on the first loss function, the second loss function, and the third loss function.

[0101] In one embodiment, the first processing unit is configured to extract global features of the first feature map from the first branch in the first model; and perform loss calculation based on the global features to obtain a first loss function.

[0102] In one embodiment, the first processing unit is configured to: identify a first local region in the second feature map that is different from other regions in the second branch of the first model using an attention mechanism, and obtain the weight corresponding to each similar point in the first local region; perform feature reconstruction on the second feature map in the third branch of the first model, and output the third feature map; obtain local features based on the weight corresponding to each similar point and the third feature map; and perform loss calculation based on the local features to obtain a second loss function.

[0103] In one embodiment, a first processing unit is configured to perform feature reconstruction on the second feature map in the third branch of the first model and output the third feature map; and to perform loss calculation based on the second feature map and the third feature map to obtain a third loss function.

[0104] According to embodiments of this disclosure, a search device is provided. Figure 8 This is a schematic diagram of the composition structure of a search device according to an embodiment of the present disclosure, such as... Figure 8 As shown, the search device 800 includes: a second input unit 801, used to input the image to be matched into a second model (the second model is a model obtained by applying the above-mentioned model training device); an extraction unit 802, used to extract local features in the image to be matched according to the second model; and a search unit 803, used to perform image search according to the local features to obtain similar images that match the image to be matched in terms of the local features.

[0105] In one embodiment, the system further includes a matching unit, which performs matching operations based on the local features to obtain a mapping matrix; and performs feature mapping based on the mapping matrix to obtain point features in the local features.

[0106] In one embodiment, the local features include: feature points matched by image pairs with similar points in the image to be matched.

[0107] In one embodiment, the matching unit is configured to form matching point pairs based on the feature points matched by image pairs with similar points in the image to be matched; and to perform the matching operation based on the matching point pairs to obtain the mapping matrix.

[0108] In one embodiment, the system further includes a quantization unit for performing PQ quantization operations based on the point features in the local features to obtain target point features; wherein the storage amount of the target point features is less than the storage amount of the point features in the local features.

[0109] In one embodiment, the quantization unit is used to acquire point features in the local features described by float; perform PQ quantization on the point features in the local features described by float to convert them into point features in the local features described by int; and use the point features in the local features described by integer int as the target point features.

[0110] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0111] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0112] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0113] like Figure 9As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 may also store various programs and data required for the operation of the electronic device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0114] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0115] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as model training methods or search methods. For example, in some embodiments, the model training method or search method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the model training method or search method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the model training method or search method by any other suitable means (e.g., by means of firmware).

[0116] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0120] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0121] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0122] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model training method, comprising: Input the sample images into the first model to be trained; In the first model, global feature extraction, local region identification, and reconstruction of the sample image are performed to obtain a target loss function. The first model includes a first branch, a second branch, and a third branch. The first branch extracts global features from a first feature map. The second branch identifies a first local region in the second feature map that differs from other regions. The third branch reconstructs features based on the second feature map to obtain a third feature map. The first local region and the third feature map are used to obtain local features. The first feature map is the feature map of the penultimate layer in the backbone network. The second feature map is the feature map of the second-to-last layer in the backbone network. The first model is trained by backpropagation of the target loss function to obtain the trained second model; The target loss function is obtained based on a first loss function, a second loss function, and a third loss function. The first loss function is an additive angular interval loss function obtained by performing additive angular interval operation on the global features of the first feature map extracted by the first branch. The second loss function is a cross-entropy loss function obtained by performing cross-entropy operation on the local features of the first local region in the second feature map identified by the second branch that is different from other regions and the third feature map output by the third branch. The third loss function is a mean squared error loss function obtained by performing mean squared error operation on the second feature map and the third feature map output by the third branch.

2. The method according to claim 1, wherein, The step of performing global feature extraction, local region recognition, and reconstruction of the sample image in the first model to obtain the target loss function includes: In the first model, at least one feature map is obtained based on the sample image; The target loss function is obtained by performing global feature extraction, local region identification, and reconstruction of the sample image based on the at least one feature map.

3. The method according to claim 2, wherein, The step of performing global feature extraction, local region identification, and sample image reconstruction based on the at least one feature map to obtain the target loss function includes: In the first branch of the first model, global features of the first feature map are extracted, and a first loss function is obtained based on the global features; The second branch in the first model identifies a first local region in the second feature map that is different from other regions, and the second loss function is obtained by combining the third feature map output by the third branch in the first model. The second feature map is combined with the third feature map output by the third branch in the first model to obtain the third loss function; The target loss function is obtained based on the first loss function, the second loss function, and the third loss function.

4. The method according to claim 3, wherein, The step of extracting global features from the first feature map in the first branch of the first model and obtaining the first loss function based on the global features includes: Extract global features from the first feature map in the first branch of the first model; The first loss function is obtained by performing loss calculations based on the global features.

5. The method according to claim 3, wherein, The step of identifying a first local region in the second feature map that differs from other regions in the second branch of the first model, and combining this with the third feature map output by the third branch of the first model to obtain a second loss function, includes: In the second branch of the first model, an attention mechanism is used to identify a first local region in the second feature map that is different from other regions, and the weights corresponding to each similar point in the first local region are obtained. In the third branch of the first model, feature reconstruction is performed on the second feature map, and the third feature map is output. Based on the weights corresponding to each similar point and the third feature map, local features are obtained; The loss function is obtained by performing loss calculations based on the local features.

6. The method according to claim 3, wherein, The step of combining the second feature map with the third feature map output by the third branch in the first model to obtain the third loss function includes: In the third branch of the first model, feature reconstruction is performed on the second feature map, and the third feature map is output. The loss function is obtained by performing loss calculations based on the second feature map and the third feature map.

7. A search method, comprising: The image to be matched is input into the second model, which is a model obtained by applying any one of the model training methods in claims 1-6 above. Based on the second model, local features are extracted from the image to be matched; Image search is performed based on the local features to obtain similar images that match the image to be matched in terms of the local features.

8. The method according to claim 7, further comprising: A matching operation is performed based on the local features to obtain a mapping matrix; Feature mapping is performed based on the mapping matrix to obtain point features in the local features.

9. The method according to claim 8, wherein, The local features include: feature points matched by image pairs with similar points in the image to be matched.

10. The method according to claim 9, wherein, The step of performing a matching operation based on the local features to obtain a mapping matrix includes: Based on the feature points matched by image pairs with similar points in the image to be matched, a matching point pair is formed; The matching operation is performed on the matching point pairs to obtain the mapping matrix.

11. The method according to any one of claims 8-10, further comprising: Based on the point features in the local features, perform product quantization (PQ) operation to obtain the target point features; The storage amount of the target point features is less than the storage amount of the point features in the local features.

12. The method according to claim 11, wherein, The step of performing product quantization (PQ) operations based on the point features in the local features to obtain the target point features includes: Obtain the point features from the local features described by floating-point numbers (float); The point features in the local features described by the float are converted into point features in the local features described by the integer int by the PQ operation. The point features in the local features described by the int are used as the target point features.

13. A model training device, comprising: The first input unit is used to input sample images into the first model to be trained. A first processing unit is configured to perform global feature extraction, local region identification, and reconstruction of the sample image within the first model to obtain a target loss function. The first model includes a first branch, a second branch, and a third branch. The first branch extracts global features from a first feature map. The second branch identifies a first local region in the second feature map that differs from other regions. The third branch reconstructs features based on the second feature map to obtain a third feature map. The first local region and the third feature map are used to obtain local features. The first feature map is the feature map of the penultimate layer in the backbone network. The second feature map is the feature map of the second-to-last layer in the backbone network. The second processing unit is used to train the first model based on the backpropagation of the target loss function to obtain the trained second model. The target loss function is obtained based on a first loss function, a second loss function, and a third loss function. The first loss function is an additive angular interval loss function obtained by performing additive angular interval operation on the global features of the first feature map extracted by the first branch. The second loss function is a cross-entropy loss function obtained by performing cross-entropy operation on the local features of the first local region in the second feature map identified by the second branch that is different from other regions and the third feature map output by the third branch. The third loss function is a mean squared error loss function obtained by performing mean squared error operation on the second feature map and the third feature map output by the third branch.

14. The apparatus according to claim 13, wherein, The first processing unit is configured to: In the first model, at least one feature map is obtained based on the sample image; The target loss function is obtained by performing global feature extraction, local region identification, and reconstruction of the sample image based on the at least one feature map.

15. The apparatus according to claim 14, wherein, The first processing unit is configured to: In the first branch of the first model, global features of the first feature map are extracted, and a first loss function is obtained based on the global features; The second branch in the first model identifies a first local region in the second feature map that is different from other regions, and the second loss function is obtained by combining the third feature map output by the third branch in the first model. The second feature map is combined with the third feature map output by the third branch in the first model to obtain the third loss function; The target loss function is obtained based on the first loss function, the second loss function, and the third loss function.

16. The apparatus according to claim 15, wherein, The first processing unit is configured to: Extract global features from the first feature map in the first branch of the first model; The first loss function is obtained by performing loss calculations based on the global features.

17. The apparatus according to claim 15, wherein, The first processing unit is configured to: In the second branch of the first model, an attention mechanism is used to identify a first local region in the second feature map that is different from other regions, and the weights corresponding to each similar point in the first local region are obtained. In the third branch of the first model, feature reconstruction is performed on the second feature map, and the third feature map is output. Based on the weights corresponding to each similar point and the third feature map, local features are obtained; The loss function is obtained by performing loss calculations based on the local features.

18. The apparatus according to claim 15, wherein, The first processing unit is used for: In the third branch of the first model, feature reconstruction is performed on the second feature map, and the third feature map is output. The loss function is obtained by performing loss calculations based on the second feature map and the third feature map.

19. A search device, comprising: The second input unit is used to input the image to be matched into the second model, wherein the second model is a model obtained by applying any one of the model training methods of claims 1-6 above. An extraction unit is used to extract local features from the image to be matched based on the second model; The search unit is used to perform image search based on the local features to obtain similar images that match the image to be matched in terms of the local features.

20. The apparatus of claim 19, further comprising a matching unit for: A matching operation is performed based on the local features to obtain a mapping matrix; Feature mapping is performed based on the mapping matrix to obtain point features in the local features.

21. The apparatus according to claim 20, wherein, The local features include: feature points matched by image pairs with similar points in the image to be matched.

22. The apparatus according to claim 21, wherein, The matching unit is used for: Based on the feature points matched by image pairs with similar points in the image to be matched, a matching point pair is formed; The matching operation is performed on the matching point pairs to obtain the mapping matrix.

23. The apparatus according to any one of claims 20-22, further comprising a quantization unit for: Based on the point features in the local features, perform product quantization (PQ) operation to obtain the target point features; in, The storage amount of the target point features is less than the storage amount of the point features in the local features.

24. The apparatus according to claim 23, wherein, The quantization unit is used for: Obtain the point features from the local features described by floating-point numbers (float); The point features in the local features described by the float are converted into point features in the local features described by the integer int by the PQ operation. The point features in the local features described by the int are used as the target point features.

25. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

27. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-12.

Images