Training method of image retrieval model, image retrieval method and device

Abstract

The application provides a training method of an image retrieval model, an image retrieval method and device, relates to the technical field of artificial intelligence such as deep learning and image processing, and the training method of the image retrieval model comprises the following steps: obtaining a plurality of sample images, each sample image is labeled with the category to which a sample object included in the sample image belongs; inputting each sample image into a target detection network in the image retrieval model to obtain a sample image block in the corresponding sample image and size information of the sample image block; inputting each sample image block and the size information into a feature extraction network in the image retrieval model to obtain a predicted feature vector of the sample object in the corresponding sample image; determining a loss value based on the predicted feature vector of each sample object and the category to which the sample object belongs, and adjusting model parameters of the target detection network and the feature extraction network based on the loss value. The training of the image retrieval model can be realized, and the image retrieval model has high retrieval accuracy when used for image retrieval.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence technology such as deep learning and image processing, and in particular to a training method for an image retrieval model, an image retrieval method, and an apparatus. Background Technology

[0002] Image retrieval technology has wide applications in various fields. For example, in e-commerce scenarios such as product recommendation, attribute management, and category merging, image retrieval technology is needed to find images similar to known images.

[0003] Image retrieval techniques in related technologies typically involve scaling the Region of Interest (ROI) of an image to a fixed size before directly inputting it into the network during the model training phase. However, since ROIs vary in size, and their dimensions reflect important attributes of the main object in the image (e.g., the aspect ratio of the ROI, i.e., the width-to-height ratio), which reflects the length of clothing, scaling the ROI to a fixed size causes distortion of the main object. Therefore, this method of directly inputting the scaled ROI into the network for model training results in the model failing to learn the true shape of the main object. Consequently, during retrieval, it is easy to retrieve images unrelated to known images, leading to poor retrieval accuracy. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art.

[0005] This application proposes a training method for an image retrieval model, an image retrieval method, and an apparatus to solve the technical problem in related technologies that image retrieval models trained in the past easily retrieve images that are unrelated to known images, resulting in poor accuracy of retrieval results.

[0006] The first aspect of this application proposes a training method for an image retrieval model, comprising: acquiring multiple sample images, each sample image being labeled with the category to which a sample object belongs; inputting each sample image into a target detection network in the image retrieval model to obtain sample image patches and size information of the corresponding sample images, wherein the sample image patches include sample objects in the sample images; inputting the sample image patches and size information of each sample image into a feature extraction network in the image retrieval model to obtain predicted feature vectors of sample objects in the corresponding sample images; determining a loss value based on the predicted feature vectors of sample objects in each sample image and the category to which the sample objects belong, and adjusting the model parameters of the target detection network and the feature extraction network based on the loss value. Thus, the image retrieval model can be trained, and the image retrieval model exhibits high retrieval accuracy when used for image retrieval.

[0007] A second aspect of this application proposes an image retrieval method, comprising: acquiring a retrieval image to be retrieved; inputting the retrieval image into a target detection network in an image retrieval model to obtain target image patches and size information of the target image patches in the retrieval image, wherein the target image patches include target objects in the retrieval image; inputting the target image patches and size information of the target image patches into a feature extraction network in the image retrieval model to obtain a predicted feature vector of the target object, wherein the image retrieval model is trained based on the method described in the first aspect embodiment; and determining a target image from multiple candidate images based on the predicted feature vector of the target object. This improves the accuracy of image retrieval.

[0008] A third aspect of this application provides a training apparatus for an image retrieval model, comprising: a first acquisition module for acquiring multiple sample images, each sample image being labeled with the category to which a sample object belongs; a first processing module for inputting each sample image into a target detection network in the image retrieval model to obtain sample image patches in the corresponding sample images and the size information of the sample image patches, the sample image patches including sample objects in the sample images; a second processing module for inputting the sample image patches in each sample image and the size information of the sample image patches into a feature extraction network in the image retrieval model to obtain predicted feature vectors of sample objects in the corresponding sample images; and a model parameter adjustment module for determining a loss value based on the predicted feature vectors of sample objects in each sample image and the category to which the sample objects belong, and adjusting the model parameters of the target detection network and the feature extraction network based on the loss value.

[0009] A fourth aspect of this application provides an image retrieval apparatus, comprising: a second acquisition module for acquiring a retrieval image to be retrieved; a third processing module for inputting the retrieval image into a target detection network in an image retrieval model to acquire target image patches in the retrieval image and size information of the target image patches, wherein the target image patches include target objects in the retrieval image; a fourth processing module for inputting the target image patches in the retrieval image and size information of the target image patches into a feature extraction network in the image retrieval model to acquire a predicted feature vector of the target object, wherein the image retrieval model is trained based on the method described in the first aspect embodiment; and a determination module for determining a target image from multiple candidate images based on the predicted feature vector of the target object.

[0010] A fifth aspect of this application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a training method for an image retrieval model as proposed in a first aspect of this application, or to perform an image retrieval method as proposed in a second aspect of this application.

[0011] A sixth aspect of this application provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform a training method for an image retrieval model as proposed in a first aspect of this application, or to perform an image retrieval method as proposed in a second aspect of this application.

[0012] A seventh aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the image retrieval model training method as proposed in the first aspect of this application, the image retrieval method as proposed in the second aspect of this application, or executes the image retrieval method as proposed in the second aspect of this application.

[0013] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0014] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0015] Figure 1 This is a flowchart illustrating the training method of the image retrieval model provided in Embodiment 1 of this application;

[0016] Figure 2 This is another schematic diagram of the training method for the image retrieval model provided in Embodiment 1 of this application;

[0017] Figure 3 This is a schematic flowchart of the training method for the image retrieval model provided in Embodiment 2 of this application;

[0018] Figure 4 This is a structural diagram of the feature extraction network provided in Embodiment 2 of this application;

[0019] Figure 5 This is a structural diagram of the first fusion module provided in Embodiment 2 of this application;

[0020] Figure 6 This is a flowchart illustrating the training method of the image retrieval model provided in Embodiment 3 of this application;

[0021] Figure 7 This is a structural diagram of the feature extraction network provided in Embodiment 3 of this application;

[0022] Figure 8 This is a schematic flowchart of the image retrieval method provided in Embodiment 4 of this application;

[0023] Figure 9 This is another schematic flowchart of the image retrieval method provided in Embodiment 4 of this application;

[0024] Figure 10 This is an example diagram illustrating the ROI processing method provided in Embodiment 4 of this application;

[0025] Figure 11 This is an example diagram of the image retrieval results provided in Embodiment 4 of this application;

[0026] Figure 12 This is a schematic diagram of the structure of the training device for the image retrieval model provided in Embodiment 5 of this application;

[0027] Figure 13 This is a schematic diagram of the image retrieval device provided in Embodiment Six of this application;

[0028] Figure 14 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present application is shown. Detailed Implementation

[0029] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0030] It should be noted that the acquisition, storage, and application of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] Image retrieval techniques in related technologies typically involve scaling the Regions of Interest (ROIs) in an image to a fixed size and then directly inputting it into the network during the model training phase. Models trained in this way are prone to retrieving images that are unrelated to known images, resulting in poor accuracy of retrieval results.

[0032] This application addresses the technical problem that image retrieval models trained in related technologies often retrieve images that are unrelated to known images, resulting in poor accuracy of retrieval results. It proposes a training method for an image retrieval model, an image retrieval method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0033] The training method for the image retrieval model includes: acquiring multiple sample images, each labeled with the category of the sample objects it contains; inputting each sample image into the object detection network of the image retrieval model to obtain sample image patches and their size information, where each sample image patch includes the sample objects; inputting the sample image patches and their size information into the feature extraction network of the image retrieval model to obtain predicted feature vectors of the sample objects in the corresponding sample images; determining a loss value based on the predicted feature vectors and the categories of the sample objects in each sample image; and adjusting the model parameters of the object detection network and the feature extraction network based on the loss value. This enables the training of the image retrieval model, which exhibits high retrieval accuracy when used for image retrieval.

[0034] The following description, with reference to the accompanying drawings, describes a training method for an image retrieval model, an image retrieval method, an apparatus, an electronic device, a storage medium, and a computer program product according to embodiments of this application.

[0035] First, the training method of the image retrieval model provided in the embodiments of this application will be described.

[0036] It should be noted that the image retrieval model training method provided in this application embodiment is executed by an image retrieval model training device. This image retrieval model training device can be an electronic device or can be configured within an electronic device to train the image retrieval model by executing the image retrieval model training method provided in this application embodiment, and the image retrieval model exhibits high retrieval accuracy when used for image retrieval.

[0037] Among them, the electronic device can be a personal computer (PC), cloud device, mobile device, server, etc. The mobile device can be any hardware device such as mobile phone, tablet computer, personal digital assistant, wearable device, vehicle device, etc. This application does not limit it.

[0038] Figure 1 This is a schematic flowchart illustrating the training method of the image retrieval model provided in Embodiment 1 of this application. Figure 1 As shown, the training method for this image retrieval model may include the following steps 101-104.

[0039] Step 101: Obtain multiple sample images, and label each sample image according to the category to which the sample objects it contains belong.

[0040] The sample images are training samples used to train the image retrieval model. The sample objects included in the sample images are the main objects in the sample images, which can be goods, people, animals, items, etc.

[0041] In the example implementation, sample images can be selected as training samples according to the needs of the application scenario. For example, when the image retrieval model is used for product recommendation in the e-commerce field, images including the products can be used as sample images to train the image retrieval model.

[0042] It should be noted that the training data used in training the image retrieval model in this application embodiment may be provided by the user and authorized for use, or obtained from public datasets, or obtained in other ways that comply with relevant laws and regulations. This application does not impose any restrictions on this.

[0043] In addition, in the embodiments of this application, when classifying sample objects in all sample images, the same sample object can be classified into one category, that is, the same sample object belongs to the same category, and different sample objects belong to different categories.

[0044] For example, consider 1000 sample images within a larger sample image set. Assume that images numbered 1-200 are images of item A taken from multiple angles, images numbered 201-400 are images of item B taken from multiple angles, images numbered 401-650 are images of item C taken from multiple angles, and images numbered 651-1000 are images of item D taken from multiple angles. Then, the 1000 sample images belong to four categories: item A belongs to category a, item B to category b, item C to category c, and item D to category d. The sample images numbered 1-200 are labeled as category a, the sample images numbered 201-400 are labeled as category b, the sample images numbered 401-650 are labeled as category c, and the sample images numbered 651-1000 are labeled as category d.

[0045] Step 102: Input each sample image into the target detection network in the image retrieval model to obtain the sample image blocks in the corresponding sample image and the size information of the sample image blocks. The sample image blocks include the sample objects in the sample image.

[0046] The image retrieval model is a neural network model with image retrieval capabilities. It includes a target detection network and a feature extraction network connected sequentially.

[0047] The object detection network is used to detect objects in any image to obtain the Region of Interest (ROI) and its size information, where the ROI includes the main object in the image. The size information of the ROI can include its height and width.

[0048] In this embodiment of the application, reference is made to Figure 2 For each sample image, the sample image can be input into the object detection network 21 in the image retrieval model. The ROI output by the object detection network 21 is the sample image patch that includes the sample object in the sample image. At the same time, the object detection network 21 can also output the size information of the sample image patch.

[0049] Step 103: Input the sample image blocks and their size information from each sample image into the feature extraction network of the image retrieval model to obtain the predicted feature vector of the sample object in the corresponding sample image.

[0050] The feature extraction network is used to extract the feature vector of the main object in any image based on the ROI and the size information of the ROI. The feature vector represents the attribute features of the main object, such as the color, pattern, style, size and other attribute features of the product.

[0051] In this embodiment of the application, reference is made to Figure 2 For each sample image, the sample image patch (ROI) and its size information can be input into the feature extraction network 22 in the image retrieval model. The feature extraction network 22 can then output the feature vector of the sample object in the sample image. In this embodiment, in order to distinguish it from other feature vectors, this feature vector is called the predicted feature vector.

[0052] Step 104: Based on the predicted feature vectors of the sample objects in each sample image and the category to which the sample objects belong, determine the loss value, and adjust the model parameters of the object detection network and the feature extraction network based on the loss value.

[0053] In the embodiments of this application, reference is made to Figure 2 The loss can be calculated based on the predicted feature vectors of sample objects in each sample image output by the feature extraction network 22. Specifically, the category of the sample object in each sample image can be predicted based on the predicted feature vectors of the sample objects in each sample image output by the feature extraction network 22. The predicted category and the labeled category are then substituted into the loss function to determine the loss value. The model parameters of the object detection network and the feature extraction network in the image retrieval model are then adjusted based on the loss value. Through multiple iterations of optimization, the trained image retrieval model is obtained.

[0054] The loss function can be set as needed, such as the cross-entropy loss function, the mean squared error (MSE) loss function, or other loss functions. This application does not impose any restrictions on this.

[0055] In this embodiment of the application, when training the image retrieval model, the sample image and the size information of the sample image blocks in the sample image are simultaneously input into the feature extraction network for model training. Compared with directly inputting the ROI of the image to the network after scaling it to a fixed size, the solution of this embodiment of the application can enable the image retrieval model to learn the real shape of the main object in the image, such as the length of the clothes, thereby improving the accuracy of the retrieval results when using the image retrieval model for image retrieval.

[0056] In summary, the image retrieval model training method provided in this application involves acquiring multiple sample images, labeling each sample image with the category of the sample objects it contains, inputting each sample image into the object detection network of the image retrieval model to obtain sample image patches and their size information. Each sample image patch includes the sample objects within the sample image. The sample image patches and their size information are then input into the feature extraction network of the image retrieval model to obtain predicted feature vectors of the sample objects in the corresponding sample images. Based on the predicted feature vectors of the sample objects in each sample image and their respective categories, a loss value is determined, and the model parameters of the object detection network and the feature extraction network are adjusted based on the loss value. This enables the training of an image retrieval model, and the image retrieval model exhibits high retrieval accuracy when used for image retrieval.

[0057] In one possible implementation, the feature extraction network may include a first size feature extraction module, an image feature extraction module, and a first fusion module connected to the first size feature extraction module and the image feature extraction module. The following section discusses the feature extraction network with the above structure, in conjunction with... Figure 3 The process of inputting the sample image blocks and their size information from each sample image into the feature extraction network of the image retrieval model in the training method of the image retrieval model in this application embodiment to obtain the predicted feature vector of the sample object in the corresponding sample image will be further explained.

[0058] Figure 3 This is a schematic flowchart illustrating the training method of the image retrieval model provided in Embodiment 2 of this application. Figure 3 As shown, the training method for the image retrieval model may include the following steps 301-306.

[0059] Step 301: Obtain multiple sample images, and label each sample image according to the category to which the sample objects it contains belong.

[0060] Step 302: Input each sample image into the target detection network in the image retrieval model to obtain the sample image blocks in the corresponding sample image and the size information of the sample image blocks. The sample image blocks include the sample objects in the sample image.

[0061] The specific implementation process and principle of steps 301-302 can be found in the descriptions of other embodiments, and will not be repeated here.

[0062] Step 303: For each sample image, input the size information of the sample image patch into the first size feature extraction module of the feature extraction network to obtain the first size feature of the sample image patch based on the size information.

[0063] refer to Figure 4 The feature extraction network may include a first size feature extraction module 41, an image feature extraction module 42, and a first fusion module 43 connected to the first size feature extraction module 41 and the image feature extraction module 42. The first size feature extraction module 41 is used to extract the size features of any image patch. The image feature extraction module 42 is used to extract the image feature vector of any image patch. The first fusion module 43 is used to fuse the size features and the image feature vector to obtain the predicted feature vector of the main object in the image.

[0064] In this embodiment of the application, for each sample image, the size information of the sample image block is input into the first size feature extraction module 41 to obtain the first size feature of the sample image block.

[0065] The size information of the sample image patch can include its width and height. Specifically, using... Represents the width of the i-th sample image patch, in... This represents the height of the i-th sample image block.

[0066] In one possible implementation, for the i-th sample image patch, the width of the sample image patch can be directly calculated. Values ​​and heights The value of is determined as the first size feature of the sample image patch. In this case, the dimension of the first size feature is 2.

[0067] In another possible implementation, for the i-th sample image patch, the ratio of the width to the height of the sample image patch can be... The aperture ratio is determined as the first size feature of the sample image patch. In this case, the dimension of the first size feature is 1.

[0068] In another possible implementation, the ratio of the width to the height of the sample image patch in each sample image can be calculated separately, and the mean of each ratio can be determined. and standard deviation Therefore, for the i-th sample image block, the first size feature of the sample image block can be determined by the following formula (1). In this case, the dimension of the first size feature is 1.

[0069] (1)

[0070] in, This represents the first size feature of the i-th sample image patch; This represents the ratio of the width to the height of the i-th sample image patch; This represents the mean of the width-to-height ratios of all sample image patches; This represents the standard deviation of the ratio of the width to the height of all sample image patches.

[0071] In another possible implementation, the logarithmic ratios of the width and height of the sample image patches in each sample image can be calculated separately, and the mean of each logarithmic ratio can be determined. and standard deviation Therefore, for the i-th sample image block, the first size feature of the i-th sample image block can be determined by the following formula (2). Here, the logarithmic ratio is the logarithm of the ratio. In this case, the dimension of the first size feature is 1.

[0072] (2)

[0073] in, This represents the first size feature of the i-th sample image patch; This represents the logarithmic ratio of the width and height of the i-th sample image patch; This represents the mean of the logarithmic ratios of the width and height of all sample image patches; The standard deviation represents the logarithmic ratio of the width and height of all sample image patches.

[0074] Step 304: Input the sample image block into the image feature extraction module included in the feature extraction network to extract image features from the sample image block and obtain the image feature vector of the sample image block.

[0075] In this embodiment of the application, a sample image block can be input into an image feature extraction module to scale the sample image block to obtain a target sample image block of a preset size, and to extract image features from the target sample image block of the preset size to obtain the image feature vector of the sample object in the sample image block.

[0076] In one possible implementation, refer to Figure 4 The image feature extraction module 42 may include an image processing submodule 421 and a first backbone network 422 connected in sequence. The image processing submodule 421 is used to scale any image patch to obtain an image patch of a preset size and obtain the tensor corresponding to the image patch. The first backbone network 422 is used to extract features from any image patch to obtain an image feature vector; it can be any network capable of feature extraction, and this application does not impose any restrictions on it.

[0077] refer to Figure 4For any sample image block, the sample image block can be input into the image processing submodule 421 to scale the sample image block to obtain a first image block of a preset size, and obtain the first tensor corresponding to the first image block. Then, the first tensor is input into the first backbone network 422 to extract image features based on the first tensor to obtain the image feature vector of the sample image block.

[0078] The preset size can be set as needed, and this application does not impose any restrictions on it. Typically, the preset size can be set to have equal width and height, for example, a preset size of 24. 24 pixels.

[0079] The first tensor may include the pixel values ​​of each pixel in the sample image block in multiple channels (such as the red channel, blue channel, and green channel).

[0080] It should be noted that, in order for the first backbone network to extract image features from the first tensor, the number of input channels of the first convolutional kernel in the first backbone network must be equal to the number of channels of the first tensor. For example, if the size of the first tensor is 24... twenty four When the number of channels in the first tensor is 3, the size of the first convolutional kernel in the first backbone network is [64, 3, 7, 7]. Here, 64 represents the number of output channels of the convolutional kernel, and 3 represents the number of input channels of the convolutional kernel.

[0081] Step 305: Input the first size feature and the image feature vector into the first fusion module connected to the first size feature extraction module and the image feature extraction module in the feature extraction network, so as to fuse the first size feature and the image feature vector to obtain the predicted feature vector of the sample object in the sample image.

[0082] In one possible implementation, refer to Figure 5 The first fusion module 43 includes a multilayer perceptron submodule 51 and a fusion submodule 52 connected in sequence. The multilayer perceptron submodule 51 can expand the dimensions of the size features to obtain size features of a preset dimension. The fusion submodule 52 can fuse the size features of the preset dimension and the image feature vector to obtain a predicted feature vector.

[0083] refer to Figure 5 For any sample image patch, the first size feature of the sample image patch can be input into the multilayer perceptron submodule 51 to expand the dimension of the first size feature and obtain the second size feature. Then, the second size feature and the image feature vector are input into the fusion submodule 52 to fuse the second size feature and the image feature vector to obtain the predicted feature vector of the sample object in the sample image.

[0084] The second-size feature and the image feature vector can be fused in several ways to obtain the predicted feature vector of the sample object in the sample image.

[0085] For example, the second-size feature and the image feature vector can be concatenated or added to obtain the predicted feature vector of the sample object in the sample image. Alternatively, the second-size feature and the image feature vector can be fused based on a self-attention mechanism to obtain the predicted feature vector of the sample object in the sample image.

[0086] For example, assuming the second size feature is a 128-dimensional feature and the image feature vector is a 2048-dimensional feature, concatenating the second size feature and the image feature vector will yield a 2176-dimensional predicted feature vector.

[0087] In the embodiments of this application, the multilayer sensing submodule 51 can be configured at different layers as shown in Table 1 below. The output feature size (i.e., dimension) is given below. The input feature size of the multilayer perceptron submodule 51 is also given. Output feature size .

[0088] Table 1. Layer design of the multilayer sensing submodule

[0089]

[0090] Step 306: Based on the predicted feature vectors of the sample objects in each sample image and the category to which the sample objects belong, determine the loss value, and adjust the model parameters of the object detection network and the feature extraction network based on the loss value.

[0091] The specific implementation process and principle of step 306 can be found in the descriptions of other embodiments, and will not be repeated here.

[0092] Therefore, it is possible to train an image retrieval model, and the image retrieval model has high retrieval accuracy when used for image retrieval.

[0093] In one possible implementation, the feature extraction network may include a second-size feature extraction module, an image processing module, a second fusion module connected to the second-size feature extraction module and the image processing module, and a second backbone network connected to the second fusion module. The following section discusses the feature extraction network with the above structure, in conjunction with... Figure 6 The process of inputting the sample image blocks and their size information from each sample image into the feature extraction network of the image retrieval model in the training method of the image retrieval model in this application embodiment to obtain the predicted feature vector of the sample object in the corresponding sample image will be further explained.

[0094] Figure 6 This is a schematic flowchart illustrating the training method of the image retrieval model provided in Embodiment 3 of this application. Figure 6 As shown, the training method for the image retrieval model may include the following steps 601-607.

[0095] Step 601: Obtain multiple sample images, and label each sample image according to the category to which the sample objects it contains belong.

[0096] Step 602: Input each sample image into the target detection network in the image retrieval model to obtain the sample image block and the size information of the sample image block in the corresponding sample image. The sample image block includes the sample object in the sample image.

[0097] The specific implementation process and principle of steps 601-602 can be found in the descriptions of other embodiments, and will not be repeated here.

[0098] Step 603: For each sample image, input the size information of the sample image block in the sample image into the second size feature extraction module included in the feature extraction network to obtain the third size feature of the sample image block based on the size information.

[0099] refer to Figure 7 The feature extraction network may include a second size feature extraction module 71, an image processing module 72, a second fusion module 73 connected to the second size feature extraction module 71 and the image processing module 72, and a second backbone network 74 connected to the second fusion module 73. The second size feature extraction module 71 is used to extract the size features of any image patch. The image processing module 72 is used to scale the arbitrary image patch to obtain an image patch of a preset size and obtain the tensor corresponding to the image patch. The second fusion module 73 is used to fuse the size features and the tensor to obtain a fused feature vector.

[0100] In this embodiment of the application, for each sample image, the size information of the sample image block is input into the second size feature extraction module 71 to obtain the third size feature of the sample image block.

[0101] The size information of the sample image block may include its width and height. The method for obtaining the third size feature of the sample image block based on the size information can refer to the method for obtaining the first size feature of the sample image block based on the size information in other embodiments; the two methods are similar and will not be repeated here.

[0102] Step 604: Input the sample image block into the image processing module included in the feature extraction network to scale the sample image block to obtain a second image block of a preset size, and obtain the second tensor corresponding to the second image block.

[0103] The preset size can be set as needed, and this application does not impose any restrictions on it. Typically, the preset size can be set to have equal width and height, for example, a preset size of 24. 24 pixels.

[0104] The second tensor can include the pixel values ​​of each pixel in the sample image block in multiple channels (such as the red channel, blue channel, and green channel).

[0105] refer to Figure 7 The sample image block is input into the image processing module 72 of the feature extraction network, which can scale the sample image block to obtain a second image block of a preset size and obtain the second tensor corresponding to the second image block.

[0106] Step 605: Input the third size feature and the second tensor into the second fusion module in the feature extraction network, which is connected to the second size feature extraction module and the image processing module, to fuse the third size feature and the second tensor to obtain a fused feature vector.

[0107] refer to Figure 7 By inputting the third size feature and the second tensor into the second fusion module 73, which is connected to the second size feature extraction module 71 and the image processing module 72 in the feature extraction network, the third size feature and the second tensor can be fused to obtain a fused feature vector.

[0108] In the embodiments of this application, the third size feature and the second tensor can be concatenated along the channel dimension to obtain a fused feature vector.

[0109] Specifically, the third-size feature and the second tensor can be concatenated along the channel dimension to obtain the fused feature vector, i.e., step 605 can be implemented in the following way:

[0110] The third size feature and the second tensor are input into the second fusion module. For each pixel in the sample image block, the third size feature is concatenated with the pixel value in multiple channels to obtain the feature vector of the pixel in multiple channels. Based on the feature vector of each pixel in the sample image block in multiple channels, a fused feature vector is generated.

[0111] For example, suppose the dimension of the third size feature is 1. The size of the second tensor is 24. twenty four 3, meaning the second tensor includes 24 samples from the image patch. The sample image patch contains 24 pixels with pixel values ​​in three channels: red, blue, and green. For each pixel, the third-size feature can be concatenated with its pixel values ​​in the three channels to obtain a feature vector in four channels. For each pixel, this feature vector includes the pixel values ​​from the red, blue, and green channels, as well as the third-size feature. Therefore, based on the 24 pixels in the sample image patch... The feature vectors of 24 pixels in 4 channels yield 24 twenty four 4-dimensional fused feature vector.

[0112] It should be noted that since the pixel values ​​of an image range from -128 to 128, and the third dimension feature is one-dimensional, its value is a decimal. If the third dimension feature in decimal form is concatenated with the pixel values ​​of the pixel in multiple channels, the resulting fused feature vector may cause fluctuations in the model. Therefore, the third dimension feature can be normalized to be close to the range of pixel values, and then concatenated with the pixel values ​​of the pixel in multiple channels to ensure the retrieval accuracy of the model.

[0113] Step 606: Input the fused feature vector into the second backbone network connected to the second fusion module in the feature extraction network to perform feature extraction based on the fused feature vector, and obtain the predicted feature vector of the sample object in the sample image.

[0114] The second backbone network is used to extract features from any feature vector to obtain a predicted feature vector. It can be any network capable of feature extraction, and this application does not impose any restrictions on it.

[0115] refer to Figure 7 By inputting the fused feature vector into the second backbone network 74 connected to the second fusion module 73 in the feature extraction network, feature extraction based on the fused feature vector can be achieved to obtain the predicted feature vector of the sample object in the sample image.

[0116] It should be noted that, in order for the second backbone network to extract features from the fused feature vector, the number of input channels of the first convolutional kernel in the second backbone network must be equal to the number of channels of the fused feature vector. For example, when the fused feature vector has 24 channels... twenty four In the 4-dimensional case, i.e., when the number of channels in the fused feature vector is 4, the size of the first convolutional kernel in the second backbone network is [64, 4, 7, 7]. Here, 64 represents the number of output channels of the convolutional kernel, and 4 represents the number of input channels of the convolutional kernel.

[0117] In the second backbone network, the first convolutional kernel can be orthogonal to initialize the weights, or other methods can be used to initialize the weights. This application does not impose any restrictions on this.

[0118] Step 607: Based on the predicted feature vectors of the sample objects in each sample image and the category to which the sample objects belong, determine the loss value, and adjust the model parameters of the object detection network and the feature extraction network based on the loss value.

[0119] The specific implementation process and principle of step 607 can be found in the descriptions of other embodiments, and will not be repeated here.

[0120] Therefore, it is possible to train an image retrieval model, and the image retrieval model has high retrieval accuracy when used for image retrieval.

[0121] Based on the training method of the image retrieval model in the above embodiments, this application provides an image retrieval method. The image retrieval method provided in this application embodiment is described below.

[0122] It should be noted that the image retrieval method provided in this application embodiment is executed by an image retrieval device. This image retrieval device can be an electronic device or can be configured within an electronic device to improve the accuracy of image retrieval by executing the image retrieval method provided in this application embodiment.

[0123] Among them, the electronic device can be a personal computer (PC), cloud device, mobile device, server, etc. The mobile device can be any hardware device such as mobile phone, tablet computer, personal digital assistant, wearable device, vehicle device, etc. This application does not limit it.

[0124] Figure 8 This is a schematic flowchart of the image retrieval method provided in Embodiment 4 of this application. Figure 8 As shown, the image retrieval method may include the following steps 801-804.

[0125] Step 801: Obtain the image to be retrieved.

[0126] The searched images include target objects, which are the main objects in the searched images and can be products, people, animals, items, etc.

[0127] Step 802: Input the retrieved image into the target detection network in the image retrieval model to obtain the target image patch in the retrieved image and the size information of the target image patch. The target image patch includes the target object in the retrieved image.

[0128] The image retrieval model is a neural network model with image retrieval functionality. It includes a target detection network and a feature extraction network connected sequentially. The image retrieval model is trained using the training method of any of the aforementioned embodiments.

[0129] The object detection network is used to detect objects in any image to obtain the Region of Interest (ROI) and its size information in the image, wherein the ROI includes the main object in the image.

[0130] In the embodiments of this application, reference is made to Figure 9 The retrieved image is input into the target detection network 21 in the image retrieval model. The ROI output by the target detection network 21 is the target image patch that contains the target object in the retrieved image. At the same time, the target detection network 21 can also output the size information of the target image patch.

[0131] The size information of the target image block can include the height and width of the target image block.

[0132] Step 803: Input the target image patch and its size information in the retrieved image into the feature extraction network of the image retrieval model to obtain the predicted feature vector of the target object.

[0133] The feature extraction network is used to extract the feature vector of the main object in any image based on the ROI and the size information of the ROI. The feature vector represents the attribute features of the main object, such as the color, pattern, style, size and other attribute features of the product.

[0134] In this embodiment of the application, reference is made to Figure 9 The target image patch and its size information in the retrieved image are input into the feature extraction network 22 in the image retrieval model. The feature extraction network 22 can then output the feature vector of the target object in the retrieved image. In this embodiment, in order to distinguish it from other feature vectors, this feature vector is called the predicted feature vector.

[0135] The specific implementation process of steps 802-803 is similar to the process of obtaining sample image blocks, sample image block size information and predicted feature vectors of sample objects in the sample image during model training, and will not be described in detail here.

[0136] Step 804: Determine the target image from multiple candidate images based on the predicted feature vector of the target object.

[0137] In one embodiment of this application, step 804 can be obtained by: acquiring candidate feature vectors of candidate objects in each candidate image; determining the similarity between the predicted feature vector of the target object and the candidate feature vectors of candidate objects in each candidate image; and determining the target image from multiple candidate images based on the similarity between the predicted feature vector of the target object and the candidate feature vectors of candidate objects in each candidate image.

[0138] The number of target images can be preset, so that the preset number of candidate images with the highest similarity to the predicted feature vector of the target object can be used as target images.

[0139] refer to Figure 9 The retrieval module can calculate the similarity between the predicted feature vector of the target object and the candidate feature vector of the candidate object in each candidate image, so as to determine the target image from multiple candidate images.

[0140] In one embodiment of this application, before obtaining the candidate feature vector of the candidate object in each candidate image, the method further includes: for each candidate image, inputting the candidate image into a target detection network to obtain candidate image blocks in the candidate image and the size information of the candidate image blocks, wherein the candidate image blocks include the candidate objects in the candidate image; inputting the candidate image blocks in the candidate image and the size information of the candidate image blocks into a feature extraction network to obtain the predicted feature vector of the candidate object.

[0141] The process of obtaining candidate image blocks, their size information, and the predicted feature vectors of candidate objects in the candidate image is similar to the process of obtaining sample image blocks, their size information, and the predicted feature vectors of sample objects in the sample image during model training, and will not be elaborated here.

[0142] Understandably, image retrieval techniques in related technologies typically scale the Regions of Interest (ROIs) in an image to a fixed-size square before directly inputting it into the network during the model training phase (hereinafter referred to as Method 1). Models trained in this way are prone to retrieving images unrelated to the known image (i.e., the retrieval image), resulting in poor retrieval accuracy. The ROI processing method of Method 1 can be found in [reference needed]. Figure 10 The third column of images.

[0143] In addition, to improve the retrieval accuracy of image retrieval models, the following methods 2, 3, and 4 are proposed. Method 2 maintains the aperture ratio of the ROI during data processing, filling the remaining portion with 0 pixels to obtain a square image. Feature extraction is then performed based on this square image, ensuring that the image input to the network reflects its aperture ratio information, thus improving the retrieval accuracy of the image retrieval model. Method 3 maintains the aperture ratio of the ROI during data processing, filling the remaining portion with the original image before retrieval to obtain a square image. Feature extraction is then performed based on this square image, ensuring that the image input to the network reflects its aperture ratio information, thus improving the retrieval accuracy of the image retrieval model. Method 4 scales the ROI in the image to a fixed-size square before directly inputting it into the network during model training. In the image retrieval stage, the aperture ratio feature is fused with the image features of the retrieval image, and the fused features are used for retrieval, thus improving the retrieval accuracy of the image retrieval model. The ROI processing method in Method 2 can be referenced. Figure 10 The fourth column of the image; the ROI processing method in Method 3 can be found in [reference needed]. Figure 10 The fifth image in the series.

[0144] Methods 2 and 3, by using 0 pixels or the original image to fill the square image used for feature extraction, reduce image resolution and lose more detailed information, resulting in inaccurate image features extracted by the model and poor retrieval accuracy. Method 4, because the feature scale of the image retrieval model differs between the training and retrieval phases, requires adjustment of the feature scale when using the trained image retrieval model for image retrieval. This debugging process is relatively cumbersome, increasing the workload for deployment, and also results in poor retrieval accuracy.

[0145] The method provided in this application uses the same feature scale for both the model training and image retrieval stages. The image retrieval model trained during the model training stage can be directly used in the image retrieval stage, thereby improving the model's retrieval accuracy while avoiding adjustments to the feature scale and reducing deployment workload. Furthermore, this application... Figure 3 and Figure 6 The image retrieval model training method provided in the illustrated embodiment, compared with method 2 and method 3, does not maintain the aperture ratio of the ROI, but scales the ROI to a preset size, thereby maximizing the preservation of image detail information and improving the retrieval accuracy of the image retrieval model.

[0146] Refer to Table 2 below and Figure 11Table 2 shows a comparison of the image retrieval results provided by the embodiment of this application with those of image retrieval methods in related technologies when performing image retrieval on a dataset of 310,000 images. Figure 11 This is a schematic diagram comparing a target image obtained by image retrieval using the image retrieval method provided in this application with a target image obtained by image retrieval using the model trained by method 1.

[0147] Table 2 Experimental Results

[0148]

[0149] According to Table 2 and Figure 11 It can be seen that the image retrieval method provided in this application embodiment can improve retrieval accuracy compared with related technologies.

[0150] In summary, the image retrieval method provided in this application obtains the image to be retrieved, inputs the retrieved image into the target detection network in the image retrieval model to obtain the target image patch and the size information of the target image patch in the retrieved image. The target image patch includes the target object in the retrieved image. The target image patch and the size information of the target image patch in the retrieved image are input into the feature extraction network in the image retrieval model to obtain the predicted feature vector of the target object. Based on the predicted feature vector of the target object, the target image is determined from multiple candidate images, thereby improving the accuracy of image retrieval.

[0151] Figure 12 This is a schematic diagram of the structure of the training device for the image retrieval model provided in Embodiment 5 of this application.

[0152] like Figure 12 As shown, the training device 1200 for the image retrieval model may include: a first acquisition module 1210, a first processing module 1220, a second processing module 1230, and a model parameter adjustment module 1240.

[0153] The first acquisition module 1210 is used to acquire multiple sample images, and each sample image is labeled with the category to which the sample objects it includes belong;

[0154] The first processing module 1220 is used to input each sample image into the target detection network in the image retrieval model to obtain the sample image block in the corresponding sample image and the size information of the sample image block. The sample image block includes the sample object in the sample image.

[0155] The second processing module 1230 is used to input the sample image blocks and the size information of the sample image blocks in each sample image into the feature extraction network in the image retrieval model in order to obtain the predicted feature vector of the sample object in the corresponding sample image.

[0156] The model parameter adjustment module 1240 is used to determine the loss value based on the predicted feature vector of the sample object in each sample image and the category to which the sample object belongs, and to adjust the model parameters of the object detection network and the feature extraction network based on the loss value.

[0157] It should be noted that the image retrieval model training device provided in this application embodiment can execute the image retrieval model training method of the foregoing embodiment. The image retrieval model training device can be an electronic device or can be configured in an electronic device to train the image retrieval model by executing the image retrieval model training method provided in this application embodiment, and the image retrieval model has high retrieval accuracy when used for image retrieval.

[0158] Among them, electronic devices can be PCs, cloud devices, mobile devices, servers, etc. Mobile devices can be any hardware devices such as mobile phones, tablets, personal digital assistants, wearable devices, and in-vehicle devices, and this application does not limit them.

[0159] In one possible implementation of this application embodiment, the second processing module 1230 includes:

[0160] The first size feature extraction unit is used to input the size information of the sample image block in each sample image into the first size feature extraction module included in the feature extraction network, so as to obtain the first size feature of the sample image block based on the size information.

[0161] The image feature extraction unit is used to input the sample image block into the image feature extraction module included in the feature extraction network to extract image features from the sample image block and obtain the image feature vector of the sample image block;

[0162] The first fusion unit is used to input the first size feature and the image feature vector into the first fusion module connected to the first size feature extraction module and the image feature extraction module in the feature extraction network, so as to fuse the first size feature and the image feature vector to obtain the predicted feature vector of the sample object in the sample image.

[0163] In one possible implementation of this application embodiment, the image feature extraction module includes an image processing submodule and a first backbone network connected in sequence;

[0164] Image feature extraction unit, used for:

[0165] Input the sample image block into the image processing submodule to scale the sample image block to obtain a first image block of a preset size, and obtain the first tensor corresponding to the first image block;

[0166] The first tensor is input into the first backbone network to extract image features based on the first tensor, thereby obtaining the image feature vector of the sample image block.

[0167] In one possible implementation of this application embodiment, the first fusion module includes a multilayer sensor submodule and a fusion submodule connected in sequence;

[0168] The first fusion unit is used for:

[0169] The first size feature is input into the multilayer perceptron submodule to expand the dimension of the first size feature and obtain the second size feature;

[0170] The second size feature and the image feature vector are input into the fusion submodule to fuse the second size feature and the image feature vector to obtain the predicted feature vector of the sample object in the sample image.

[0171] In one possible implementation of this application embodiment, the size information includes the width and height of the sample image block; the first size feature extraction unit is used for:

[0172] The width and height of the sample image block are determined as the first size feature;

[0173] Alternatively, the ratio of the width to the height of the sample image patch can be determined as the first size feature;

[0174] Alternatively, determine the ratio of the width to the height of the sample image patch in each sample image, and determine the mean and standard deviation of each ratio. Based on the ratio, mean, and standard deviation, determine the first size feature.

[0175] Alternatively, determine the logarithmic ratio of the width and height of the sample image blocks in each sample image, and determine the mean and standard deviation of each logarithmic ratio. Based on the logarithmic ratio, the mean and standard deviation of each logarithmic ratio, determine the first size feature.

[0176] In one possible implementation of this application embodiment, the second processing module 1230 includes:

[0177] The second size feature extraction unit is used to input the size information of the sample image block in the sample image into the second size feature extraction module included in the feature extraction network for each sample image, so as to obtain the third size feature of the sample image block based on the size information;

[0178] The image processing unit is used to input the sample image block into the image processing module included in the feature extraction network to scale the sample image block to obtain a second image block of a preset size, and to obtain the second tensor corresponding to the second image block;

[0179] The second fusion unit is used to input the third size feature and the second tensor into the second fusion module in the feature extraction network, which is connected to the second size feature extraction module and the image processing module, so as to fuse the third size feature and the second tensor to obtain a fused feature vector.

[0180] The fusion feature extraction unit is used to input the fusion feature vector into the second backbone network connected to the second fusion module in the feature extraction network, so as to perform feature extraction based on the fusion feature vector and obtain the predicted feature vector of the sample object in the sample image.

[0181] In one possible implementation of this application embodiment, the second tensor includes the pixel values ​​of each pixel in the sample image block in multiple channels;

[0182] The second fusion unit is used for:

[0183] The third size feature and the second tensor are input into the second fusion module so that for each pixel in the sample image block, the third size feature is concatenated with the pixel value in multiple channels to obtain the feature vector of the pixel in multiple channels.

[0184] A fused feature vector is generated based on the feature vectors of each pixel in the sample image block across multiple channels.

[0185] It should be noted that the explanations in the aforementioned training method embodiments of the image retrieval model also apply to the training device of the image retrieval model in this embodiment, and will not be repeated here.

[0186] The image retrieval model training apparatus of this application embodiment acquires multiple sample images, each sample image is labeled with the category of the sample objects it contains, and each sample image is input into the object detection network in the image retrieval model to obtain sample image patches and their size information. Each sample image patch includes the sample objects in the sample image. The sample image patches and their size information are then input into the feature extraction network in the image retrieval model to obtain predicted feature vectors of the sample objects in the corresponding sample images. Based on the predicted feature vectors of the sample objects in each sample image and their categories, a loss value is determined, and the model parameters of the object detection network and the feature extraction network are adjusted based on the loss value. Therefore, the image retrieval model can be trained, and this image retrieval model has high retrieval accuracy when used for image retrieval.

[0187] Figure 13 This is a schematic diagram of the image retrieval device provided in Embodiment Six of this application.

[0188] like Figure 13As shown, the image retrieval device 1300 may include: a second acquisition module 1310, a third processing module 1320, a fourth processing module 1330, and a determination module 1340.

[0189] The second acquisition module 1310 is used to acquire the search image to be retrieved;

[0190] The third processing module 1320 is used to input the retrieved image into the target detection network in the image retrieval model to obtain the target image block in the retrieved image and the size information of the target image block, wherein the target image block includes the target object in the retrieved image.

[0191] The fourth processing module 1330 is used to input the target image patch and the size information of the target image patch in the retrieved image into the feature extraction network in the image retrieval model to obtain the predicted feature vector of the target object. The image retrieval model is trained based on the training method of any of the aforementioned image retrieval models.

[0192] The determination module 1340 is used to determine the target image from multiple candidate images based on the predicted feature vector of the target object.

[0193] In one possible implementation of this application embodiment, the determining module 1340 includes:

[0194] The acquisition unit is used to acquire the candidate feature vectors of candidate objects in each candidate image;

[0195] The first determining unit is used to determine the similarity between the predicted feature vector of the target object and the candidate feature vector of the candidate object in each candidate image;

[0196] The second determining unit is used to determine the target image from multiple candidate images based on the similarity between the predicted feature vector of the target object and the candidate feature vector of the candidate object in each candidate image.

[0197] In one possible implementation of this application embodiment, the image retrieval device 1300 may further include:

[0198] The third acquisition module is used to input the candidate image into the object detection network for each candidate image to obtain the candidate image blocks in the candidate image and the size information of the candidate image blocks. The candidate image blocks include the candidate objects in the candidate image.

[0199] The fifth processing module is used to input the candidate image patches and their size information into the feature extraction network to obtain the predicted feature vectors of the candidate objects.

[0200] It should be noted that the image retrieval device provided in this application embodiment can execute the image retrieval method of the foregoing embodiment. The image retrieval device can be an electronic device or can be configured in an electronic device to improve the accuracy of image retrieval by executing the image retrieval method provided in this application embodiment.

[0201] Among them, electronic devices can be PCs, cloud devices, mobile devices, servers, etc. Mobile devices can be any hardware devices such as mobile phones, tablets, personal digital assistants, wearable devices, and in-vehicle devices, and this application does not limit them.

[0202] It should be noted that the explanations and descriptions in the foregoing image retrieval method embodiments also apply to the image retrieval device of this embodiment, and will not be repeated here.

[0203] The image retrieval device of this application embodiment acquires a retrieval image to be retrieved, inputs the retrieval image into a target detection network in an image retrieval model to obtain target image blocks and size information of the target image blocks in the retrieval image. The target image blocks include target objects in the retrieval image. The target image blocks and size information of the target image blocks in the retrieval image are input into a feature extraction network in an image retrieval model to obtain a predicted feature vector of the target object. Based on the predicted feature vector of the target object, the target image is determined from multiple candidate images, thereby improving the accuracy of image retrieval.

[0204] To implement the above embodiments, this application also proposes an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a training method for an image retrieval model as proposed in any of the foregoing embodiments of this application, or to perform an image retrieval method as proposed in any of the foregoing embodiments of this application.

[0205] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing computer instructions, which are used to cause the computer to execute a training method for an image retrieval model as proposed in any of the foregoing embodiments of this application, or to execute an image retrieval method as proposed in any of the foregoing embodiments of this application.

[0206] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements a training method for an image retrieval model as proposed in any of the foregoing embodiments of this application, or implements an image retrieval method as proposed in any of the foregoing embodiments of this application.

[0207] Figure 14A block diagram of an exemplary electronic device suitable for implementing embodiments of the present application is shown. Figure 14 The electronic device 1400 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0208] like Figure 14 As shown, the electronic device 1400 is presented in the form of a general-purpose computing device. The components of the electronic device 1400 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0209] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0210] Electronic device 1400 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 1400, including volatile and non-volatile media, removable and non-removable media.

[0211] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Electronic device 1400 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 14 Not shown; usually referred to as a "hard drive"). Although Figure 14As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0212] A program / utility 40 having a set (at least one) of program modules 1442 may be stored, for example, in memory 28. Such program modules 1442 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 1442 typically perform the functions and / or methods described in the embodiments of this application.

[0213] Electronic device 1400 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 1400, and / or with any device that enables electronic device 1400 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 1422. Furthermore, electronic device 1400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 1400 via bus 18. It should be understood that, although... Figure 14 As not shown, other hardware and / or software modules may be used in conjunction with the electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0214] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the methods mentioned in the foregoing embodiments.

[0215] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0216] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0217] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0218] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0219] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0220] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0221] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0222] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A training method for an image retrieval model, characterized in that, The method includes: Multiple sample images are acquired, and each sample image is labeled with the category to which the sample objects it contains belong; Each of the sample images is input into the target detection network in the image retrieval model to obtain sample image blocks in the corresponding sample image and the size information of the sample image blocks, wherein the sample image blocks include sample objects in the sample image; The sample image blocks and their size information in each of the sample images are input into the feature extraction network of the image retrieval model to obtain the predicted feature vector of the sample object in the corresponding sample image. Based on the predicted feature vectors of the sample objects in each sample image and the category to which the sample objects belong, a loss value is determined, and the model parameters of the target detection network and the feature extraction network are adjusted based on the loss value. The step of inputting the sample image patches and their size information from each of the sample images into the feature extraction network of the image retrieval model to obtain the predicted feature vector of the sample object in the corresponding sample image includes: For each of the sample images, the size information of the sample image block in the sample image is input into the second size feature extraction module included in the feature extraction network, so as to obtain the third size feature of the sample image block based on the size information; The sample image block is input into the image processing module included in the feature extraction network to scale the sample image block to obtain a second image block of a preset size, and the second tensor corresponding to the second image block is obtained. The third size feature and the second tensor are input into the second fusion module in the feature extraction network, which is connected to the second size feature extraction module and the image processing module, to fuse the third size feature and the second tensor to obtain a fused feature vector; The fused feature vector is input into the second backbone network connected to the second fusion module in the feature extraction network to perform feature extraction based on the fused feature vector, thereby obtaining the predicted feature vector of the sample object in the sample image. The second tensor includes the pixel values ​​of each pixel in the sample image block across multiple channels; The step of inputting the third size feature and the second tensor into the second fusion module in the feature extraction network, which is connected to the second size feature extraction module and the image processing module, to fuse the third size feature and the second tensor to obtain a fused feature vector includes: The third size feature and the second tensor are input into the second fusion module so that for each pixel in the sample image block, the third size feature is concatenated with the pixel value of the pixel in the multiple channels to obtain the feature vector of the pixel in the multiple channels. The fused feature vector is generated based on the feature vectors of each pixel in the sample image block across multiple channels.

2. The method according to claim 1, characterized in that, The step of inputting the sample image patches and their size information from each of the sample images into the feature extraction network of the image retrieval model to obtain the predicted feature vector of the sample object in the corresponding sample image includes: For each of the sample images, the size information of the sample image blocks is input into the first size feature extraction module of the feature extraction network to obtain the first size feature of the sample image block based on the size information; The sample image block is input into the image feature extraction module included in the feature extraction network to extract image features from the sample image block and obtain the image feature vector of the sample image block; The first size feature and the image feature vector are input into the first fusion module in the feature extraction network, which is connected to the first size feature extraction module and the image feature extraction module, to fuse the first size feature and the image feature vector to obtain the predicted feature vector of the sample object in the sample image.

3. The method according to claim 2, characterized in that, The image feature extraction module includes an image processing sub-module and a first backbone network connected in sequence; The step of inputting the sample image block into the feature extraction network, including the image feature extraction module, to extract image features from the sample image block and obtain the image feature vector of the sample image block, includes: The sample image block is input into the image processing submodule to scale the sample image block to obtain a first image block of a preset size, and the first tensor corresponding to the first image block is obtained. The first tensor is input into the first backbone network to extract image features based on the first tensor, thereby obtaining the image feature vector of the sample image block.

4. The method according to claim 2, characterized in that, The first fusion module includes a multilayer sensor submodule and a fusion submodule connected in sequence; The step of inputting the first size feature and the image feature vector into a first fusion module connected to the first size feature extraction module and the image feature extraction module in the feature extraction network to fuse the first size feature and the image feature vector to obtain a predicted feature vector of the sample object in the sample image includes: The first size feature is input into the multilayer perceptron submodule to expand the dimension of the first size feature and obtain the second size feature; The second size feature and the image feature vector are input into the fusion submodule to fuse the second size feature and the image feature vector to obtain the predicted feature vector of the sample object in the sample image.

5. The method according to any one of claims 2-4, characterized in that, The size information includes the width and height of the sample image block; obtaining the first size feature of the sample image block based on the size information includes: The width and height of the sample image block are determined as the first size feature; Alternatively, the ratio of the width to the height of the sample image block can be determined as the first size feature; Alternatively, determine the ratio of the width to the height of the sample image blocks in each of the sample images, and determine the mean and standard deviation of each ratio; based on the ratio, the mean, and the standard deviation, determine the first size feature. Alternatively, determine the logarithmic ratio of the width and height of the sample image blocks in each of the sample images, and determine the mean and standard deviation of each of the logarithmic ratios. Based on the logarithmic ratios, the mean and standard deviation of each of the logarithmic ratios, determine the first size feature.

6. An image retrieval method, characterized in that, The method includes: Obtain the image to be retrieved; The retrieved image is input into the target detection network in the image retrieval model to obtain target image patches in the retrieved image and the size information of the target image patches, wherein the target image patches include the target objects in the retrieved image; The target image patch and its size information in the retrieved image are input into the feature extraction network of the image retrieval model to obtain the predicted feature vector of the target object, wherein the image retrieval model is trained based on the method of any one of claims 1-5; The target image is determined from multiple candidate images based on the predicted feature vector of the target object.

7. The method according to claim 6, characterized in that, The step of determining the target image from multiple candidate images based on the predicted feature vector of the target object includes: Obtain the candidate feature vectors of the candidate objects in each of the candidate images; Determine the similarity between the predicted feature vector of the target object and the candidate feature vectors of candidate objects in each of the candidate images; The target image is determined from the plurality of candidate images based on the similarity between the predicted feature vector of the target object and the candidate feature vector of the candidate object in each of the candidate images.

8. The method according to claim 7, characterized in that, Before obtaining the candidate feature vectors of candidate objects in each of the candidate images, the method further includes: For each candidate image, the candidate image is input into the target detection network to obtain candidate image blocks in the candidate image and the size information of the candidate image blocks, wherein the candidate image blocks include candidate objects in the candidate image; The candidate image patches and their size information are input into the feature extraction network to obtain the predicted feature vector of the candidate object.

9. A training device for an image retrieval model, characterized in that, The device includes: The first acquisition module is used to acquire multiple sample images, each of which is labeled with the category to which the sample object it includes belongs; The first processing module is used to input each of the sample images into the target detection network in the image retrieval model to obtain sample image blocks in the corresponding sample images and the size information of the sample image blocks, wherein the sample image blocks include sample objects in the sample images; The second processing module is used to input the sample image blocks and the size information of the sample image blocks in each of the sample images into the feature extraction network in the image retrieval model in order to obtain the predicted feature vector of the sample object in the corresponding sample image. The model parameter adjustment module is used to determine the loss value based on the predicted feature vector of the sample object in each sample image and the category to which the sample object belongs, and to adjust the model parameters of the target detection network and the feature extraction network based on the loss value; The second processing module includes: The second size feature extraction unit is used to input the size information of the sample image block in the sample image into the second size feature extraction module included in the feature extraction network for each sample image, so as to obtain the third size feature of the sample image block based on the size information; An image processing unit is used to input the sample image block into the image processing module included in the feature extraction network to scale the sample image block to obtain a second image block of a preset size, and to obtain a second tensor corresponding to the second image block; The second fusion unit is used to input the third size feature and the second tensor into the second fusion module in the feature extraction network, which is connected to the second size feature extraction module and the image processing module, so as to fuse the third size feature and the second tensor to obtain a fused feature vector; The fusion feature extraction unit is used to input the fusion feature vector into the second backbone network connected to the second fusion module in the feature extraction network, so as to perform feature extraction based on the fusion feature vector and obtain the predicted feature vector of the sample object in the sample image. The second tensor includes the pixel values ​​of each pixel in the sample image block across multiple channels; The second fusion unit is used for: The third size feature and the second tensor are input into the second fusion module so that for each pixel in the sample image block, the third size feature is concatenated with the pixel value of the pixel in the multiple channels to obtain the feature vector of the pixel in the multiple channels. The fused feature vector is generated based on the feature vectors of each pixel in the sample image block across multiple channels.

10. The apparatus according to claim 9, characterized in that, The second processing module includes: The first size feature extraction unit is used to input the size information of the sample image block in each sample image into the first size feature extraction module included in the feature extraction network, so as to obtain the first size feature of the sample image block based on the size information. An image feature extraction unit is used to input the sample image block into the image feature extraction module included in the feature extraction network to extract image features from the sample image block and obtain the image feature vector of the sample image block; The first fusion unit is used to input the first size feature and the image feature vector into the first fusion module connected to the first size feature extraction module and the image feature extraction module in the feature extraction network, so as to fuse the first size feature and the image feature vector to obtain the predicted feature vector of the sample object in the sample image.

11. The apparatus according to claim 10, characterized in that, The image feature extraction module includes an image processing sub-module and a first backbone network connected in sequence; The image feature extraction unit is used for: The sample image block is input into the image processing submodule to scale the sample image block to obtain a first image block of a preset size, and the first tensor corresponding to the first image block is obtained. The first tensor is input into the first backbone network to extract image features based on the first tensor, thereby obtaining the image feature vector of the sample image block.

12. The apparatus according to claim 10, characterized in that, The first fusion module includes a multilayer sensor submodule and a fusion submodule connected in sequence; The first fusion unit is used for: The first size feature is input into the multilayer perceptron submodule to expand the dimension of the first size feature and obtain the second size feature; The second size feature and the image feature vector are input into the fusion submodule to fuse the second size feature and the image feature vector to obtain the predicted feature vector of the sample object in the sample image.

13. The apparatus according to any one of claims 10-12, characterized in that, The size information includes the width and height of the sample image block; the first size feature extraction unit is used for: The width and height of the sample image block are determined as the first size feature; Alternatively, the ratio of the width to the height of the sample image block can be determined as the first size feature; Alternatively, determine the ratio of the width to the height of the sample image blocks in each of the sample images, and determine the mean and standard deviation of each ratio; based on the ratio, the mean, and the standard deviation, determine the first size feature. Alternatively, determine the logarithmic ratio of the width and height of the sample image blocks in each of the sample images, and determine the mean and standard deviation of each of the logarithmic ratios. Based on the logarithmic ratios, the mean and standard deviation of each of the logarithmic ratios, determine the first size feature.

14. An image retrieval device, characterized in that, The device includes: The second acquisition module is used to acquire the search image to be retrieved; The third processing module is used to input the retrieved image into the target detection network in the image retrieval model to obtain the target image block in the retrieved image and the size information of the target image block, wherein the target image block includes the target object in the retrieved image; The fourth processing module is used to input the target image patch and the size information of the target image patch in the retrieved image into the feature extraction network of the image retrieval model to obtain the predicted feature vector of the target object, wherein the image retrieval model is trained based on the method of any one of claims 1-5; The determination module is used to determine the target image from multiple candidate images based on the predicted feature vector of the target object.

15. The apparatus according to claim 14, characterized in that, The determining module includes: The acquisition unit is used to acquire the candidate feature vectors of candidate objects in each of the candidate images; The first determining unit is used to determine the similarity between the predicted feature vector of the target object and the candidate feature vector of the candidate object in each of the candidate images; The second determining unit is used to determine the target image from the plurality of candidate images based on the similarity between the predicted feature vector of the target object and the candidate feature vector of the candidate object in each of the candidate images.

16. The apparatus according to claim 15, characterized in that, Also includes: The third acquisition module is used to input the candidate image into the target detection network for each candidate image to obtain candidate image blocks in the candidate image and the size information of the candidate image blocks, wherein the candidate image blocks include candidate objects in the candidate image; The fifth processing module is used to input the candidate image blocks in the candidate image and the size information of the candidate image blocks into the feature extraction network to obtain the predicted feature vector of the candidate object.

17. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-5, or the method of any one of claims 6-8.

18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method of any one of claims 1-5, or to perform the method of any one of claims 6-8.

Images