Feature extraction model training method, device, equipment and storage medium

Abstract

The embodiments of the present application disclose a method, device, equipment and storage medium for training a feature extraction model, and belong to the field of computer technology. The method includes: obtaining sample images and sample category features; calling a feature extraction model to extract features from the sample image to obtain image features; dividing the image features into semantic features and non-semantic features; based on the first difference between the semantic features and the sample category features, training the feature extraction model so that the difference between the semantic features and the sample category features divided by the trained feature extraction model is reduced. Since the feature extraction model can not only learn the semantic features related to the category in the image, but also does not lose the non-semantic features in the image that are not related to the category, the image features extracted by the feature extraction model can fully characterize the features of the image, thereby improving the accuracy of the image features extracted by the feature extraction model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a training method, apparatus, device and storage medium for a feature extraction model. Background Art

[0002] With the rapid development of computer technology, image features are widely used in various fields such as image search and image recognition, and have important research significance.

[0003] In related technologies, a feature extraction model is usually trained based on similar images so that the feature extraction model can learn the correlation between image features of similar images, thereby having the ability to extract image features. The trained feature extraction model can be used to extract image features of any image.

[0004] However, since only the similarity between images is considered when training the feature extraction model, the image features extracted by the feature extraction model are difficult to fully characterize the features of the entire image, resulting in the extracted image features being inaccurate. Summary of the invention

[0005] The embodiment of the present application provides a method, apparatus, device and storage medium for training a feature extraction model, which can improve the accuracy of image features extracted by the feature extraction model. The technical solution is as follows:

[0006] In one aspect, a method for training a feature extraction model is provided, the method comprising:

[0007] Acquire a sample image and a sample category feature, wherein the sample category feature represents the category to which the sample image belongs;

[0008] Calling a feature extraction model to extract features from the sample image to obtain image features;

[0009] Dividing the image features into semantic features and non-semantic features, wherein the semantic features represent the semantics of the sample image predicted by the feature extraction model, and the non-semantic features are other features of the image features except the semantic features;

[0010] Based on the first difference between the semantic feature and the sample category feature, the feature extraction model is trained so that the difference between the semantic feature and the sample category feature divided by the trained feature extraction model is reduced.

[0011] Optionally, generating a plurality of category features includes:

[0012] generating a Hadamard matrix, wherein the Hadamard matrix includes a plurality of Hadamard vectors;

[0013] The other Hadamard vectors except the Hadamard vectors in the first row in the Hadamard matrix are determined as the category features.

[0014] Optionally, the training the feature extraction model based on the first difference between the semantic feature and the sample category feature includes:

[0015] determining a first loss parameter, wherein the first loss parameter is positively correlated with the first difference;

[0016] Based on the first loss parameter, the feature extraction model is trained.

[0017] Optionally, the training the feature extraction model based on the first difference and a second difference between the image feature and the quantitative feature includes:

[0018] determining a first loss parameter, wherein the first loss parameter is positively correlated with the first difference;

[0019] determining a third loss parameter, wherein the third loss parameter is positively correlated with the second difference;

[0020] The feature extraction model is trained based on the first loss parameter and the third loss parameter.

[0021] Optionally, the training the feature extraction model based on at least one first difference, a third difference, and a fourth difference includes:

[0022] determining a first loss parameter, the first loss parameter being positively correlated with the at least one first difference;

[0023] Determine a fourth loss parameter, wherein the fourth loss parameter is negatively correlated with a fifth difference, and the fifth difference is a difference between the third difference and the fourth difference;

[0024] The feature extraction model is trained based on the first loss parameter and the fourth loss parameter.

[0025] Optionally, respectively determining the similarity between the first sample image and each of the candidate sample images includes:

[0026] Calling the feature extraction model to perform feature extraction on the first sample image and each of the candidate sample images respectively, to obtain a first image feature of the first sample image and a second image feature of each of the candidate sample images;

[0027] The similarity between the first image feature and each of the second image features is determined respectively.

[0028] Optionally, the acquiring the first sample image, the second sample image, and a plurality of candidate sample images includes:

[0029] Acquire multiple sample pairs, each sample pair including two similar sample images;

[0030] Determine two sample images in a target sample pair as the first sample image and the second sample image respectively, wherein the target sample pair is any sample pair among the multiple sample pairs;

[0031] The sample images in the other sample pairs among the multiple sample pairs except the target sample pair are determined as the candidate sample images.

[0032] Optionally, the image features in the image feature library are binary quantized features, and searching the image feature library for a central image feature whose similarity with the third image feature satisfies a first similarity condition includes:

[0033] Performing binary quantization on the third image feature to obtain a target quantized feature;

[0034] In the image feature library, a central image feature whose similarity with the target quantitative feature satisfies the first similarity condition is searched.

[0035] Optionally, searching the image feature library for a central image feature whose similarity with the third image feature satisfies a first similarity condition includes:

[0036] Searching the image feature library for a third number of central image features having the greatest similarity to the third image feature;

[0037] The step of searching, in the determined feature set, an image feature whose similarity with the third image feature satisfies a second similarity condition, comprises:

[0038] In the determined feature set, a fourth number of image features having the greatest similarity to the third image feature are searched.

[0039] On the other hand, a training device for a feature extraction model is provided, the device comprising:

[0040] A sample acquisition module, used to acquire a sample image and a sample category feature, wherein the sample category feature represents the category to which the sample image belongs;

[0041] A feature extraction module is used to call a feature extraction model to extract features from the sample image to obtain image features;

[0042] A feature division module, used for dividing the image features into semantic features and non-semantic features, wherein the semantic features represent the semantics of the sample image predicted by the feature extraction model, and the non-semantic features are other features of the image features except the semantic features;

[0043] The model training module is used to train the feature extraction model based on the first difference between the semantic feature and the sample category feature, so that the difference between the semantic feature divided by the trained feature extraction model and the sample category feature is reduced.

[0044] Optionally, the feature division module includes:

[0045] A mask acquisition unit, used to acquire a semantic mask and a non-semantic mask, wherein the semantic mask indicates the position of the semantic feature of any image in the image feature, and the non-semantic mask indicates the position of the non-semantic feature of any image in the image feature;

[0046] A mask processing unit, configured to perform mask processing on the image feature based on the semantic mask, and determine the obtained feature as the semantic feature;

[0047] The mask processing unit is used to perform mask processing on the image feature based on the non-semantic mask, and determine the obtained feature as the non-semantic feature.

[0048] Optionally, the feature division module includes:

[0049] A relationship acquisition unit, used to acquire a mapping relationship, wherein the mapping relationship refers to a mapping relationship between a position in an image feature of any image and a position in a class feature;

[0050] A position determination unit is used to determine, based on the mapping relationship, a plurality of second positions in the image feature that have a mapping relationship with a plurality of first positions in the sample category feature, wherein the features at the plurality of second positions in the image feature constitute the semantic feature, and the other features in the image feature except the features at the plurality of second positions constitute the non-semantic feature.

[0051] Optionally, the sample acquisition module includes:

[0052] A generating unit, configured to generate a plurality of category features, each category feature comprising a plurality of values, and the number of each type of values ​​in the plurality of values ​​is equal;

[0053] A classification unit, used to assign the plurality of category features to a plurality of preset categories respectively, each preset category corresponding to a different category feature;

[0054] A first determining unit, configured to determine the multiple categories to which the sample image belongs;

[0055] The second determining unit is configured to form the sample category feature from category features corresponding to the multiple categories.

[0056] Optionally, the generating unit is used to:

[0057] generating a Hadamard matrix, wherein the Hadamard matrix includes a plurality of Hadamard vectors;

[0058] The other Hadamard vectors except the Hadamard vectors in the first row in the Hadamard matrix are determined as the category features.

[0059] Optionally, the model training module includes:

[0060] a loss determination unit, configured to determine a first loss parameter, wherein the first loss parameter is positively correlated with the first difference;

[0061] A training unit is used to train the feature extraction model based on the first loss parameter.

[0062] Optionally, the model training module is used to:

[0063] Based on the first difference and the uncertainty parameter of the non-semantic feature, the feature extraction model is trained so that the difference between the semantic feature divided by the trained feature extraction model and the sample category feature is reduced and the uncertainty parameter of the divided non-semantic feature is increased, wherein the uncertainty parameter represents the degree of uncertainty of the non-semantic feature.

[0064] Optionally, the model training module includes:

[0065] a loss determination unit, configured to determine a first loss parameter, wherein the first loss parameter is positively correlated with the first difference;

[0066] The loss determination unit is further used to determine a second loss parameter, where the second loss parameter is negatively correlated with the uncertainty parameter;

[0067] A training unit is used to train the feature extraction model based on the first loss parameter and the second loss parameter.

[0068] Optionally, the model training module is used to:

[0069] Performing binary quantization on the image features to obtain quantized features;

[0070] Based on the first difference and the second difference between the image feature and the quantitative feature, the feature extraction model is trained so that the difference between the semantic feature divided by the trained feature extraction model and the sample category feature is reduced, and the difference between the image feature obtained by the trained feature extraction model and the corresponding quantitative feature is reduced.

[0071] Optionally, the model training module includes:

[0072] a loss determination unit, configured to determine a first loss parameter, wherein the first loss parameter is positively correlated with the first difference;

[0073] The loss determination unit is further used to determine a third loss parameter, where the third loss parameter is positively correlated with the second difference;

[0074] A training unit is used to train the feature extraction model based on the first loss parameter and the third loss parameter.

[0075] Optionally, the sample image includes a first sample image, a second sample image and a third sample image, the second sample image is similar to the first sample image, and the third sample image is not similar to the first sample image;

[0076] The model training module is used to:

[0077] Based on at least one first difference, a third difference, and a fourth difference, the feature extraction model is trained so that the difference between the semantic features divided by the trained feature extraction model and the sample category features is reduced, and the difference between the image features of dissimilar images obtained by the trained feature extraction model is greater than the difference between the image features of similar images;

[0078] Among them, the at least one first difference value is the difference between the semantic features and the sample category features of at least one image among the first sample image, the second sample image and the third sample image, the third difference value is the difference between the image features of the first sample image and the image features of the second sample image, and the fourth difference value is the difference between the image features of the first sample image and the image features of the third sample image.

[0079] Optionally, the model training module includes:

[0080] a loss determination unit, configured to determine a first loss parameter, wherein the first loss parameter is positively correlated with the at least one first difference;

[0081] The loss determination unit is further used to determine a fourth loss parameter, wherein the fourth loss parameter is negatively correlated with a fifth difference, and the fifth difference is a difference between the third difference and the fourth difference;

[0082] A training unit is used to train the feature extraction model based on the first loss parameter and the fourth loss parameter.

[0083] Optionally, the sample acquisition module includes:

[0084] An image acquisition unit, configured to acquire the first sample image, the second sample image, and a plurality of candidate sample images;

[0085] a similarity determination unit, configured to respectively determine the similarity between the first sample image and each of the candidate sample images;

[0086] The image determination unit is used to delete the largest first number of similarities among the determined multiple similarities, and determine the candidate sample images corresponding to the largest second number of similarities among the remaining multiple similarities as sample images that are not similar to the first sample image.

[0087] Optionally, the image acquisition unit is used to:

[0088] Acquire multiple sample pairs, each sample pair including two similar sample images;

[0089] Determine two sample images in a target sample pair as the first sample image and the second sample image respectively, wherein the target sample pair is any sample pair among the multiple sample pairs;

[0090] The sample images in the other sample pairs among the multiple sample pairs except the target sample pair are determined as the candidate sample images.

[0091] Optionally, the device further comprises:

[0092] A retrieval module, configured to respond to a retrieval instruction and obtain a target image carried in the retrieval instruction;

[0093] The feature extraction module is further used to call the trained feature extraction model to perform feature extraction on the target image to obtain a third image feature of the target image;

[0094] The retrieval module is further used to search for similar images of the target image based on the third image feature.

[0095] Optionally, the image feature library includes a plurality of feature sets, each of which includes a central image feature and image features similar to the central image feature;

[0096] The retrieval module comprises:

[0097] A feature retrieval unit, configured to search the image feature library for a central image feature whose similarity with the third image feature satisfies a first similarity condition;

[0098] A set determination unit, used to determine the feature set where the found central image feature belongs;

[0099] The image retrieval unit is used to search for an image feature whose similarity with the third image feature satisfies a second similarity condition in the determined feature set, and determine the image corresponding to the searched image feature as a similar image to the target image.

[0100] Optionally, the image features in the image feature library are binary quantized features, and the feature retrieval unit is used to:

[0101] Binary quantization is performed on the third image feature to obtain a target quantized feature;

[0102] In the image feature library, a central image feature whose similarity with the target quantitative feature satisfies the first similarity condition is searched.

[0103] Optionally, the feature retrieval unit is used to: search for a third number of central image features having the greatest similarity with the third image feature in the image feature library;

[0104] The image retrieval unit is used to search for a fourth number of image features having the greatest similarity with the third image feature in the determined feature set.

[0105] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the training method of the feature extraction model as described in the above aspects.

[0106] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the training method of the feature extraction model as described in the above aspects.

[0107] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is loaded and executed by a processor to implement the operations performed by the training method of the feature extraction model as described in the above aspects.

[0108] The method, apparatus, device and storage medium provided in the embodiments of the present application divide the image features extracted by the feature extraction model into semantic features and non-semantic features, wherein the semantic features represent the features in the image that are related to semantics, and the non-semantic features represent the features in the image that are not related to semantics, and train the feature extraction model through the difference between the semantic features of the sample image and the corresponding sample category features, so that the feature extraction model can not only learn the semantic features in the image that are related to the category, but also will not lose the non-semantic features in the image that are not related to the category. Therefore, the image features extracted by the feature extraction model trained by the method can fully characterize the features of the image, thereby improving the accuracy of the image features extracted by the feature extraction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0110] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0111] Figure 2 It is a flow chart of a method for training a feature extraction model provided in an embodiment of the present application;

[0112] Figure 3 is a schematic diagram of a mapping relationship provided in an embodiment of the present application;

[0113] Figure 4 is a schematic diagram of a method for training a feature extraction model provided in an embodiment of the present application;

[0114] Figure 5 is a schematic diagram of a feature space provided in an embodiment of the present application;

[0115] Figure 6 is a flow chart of a sample image acquisition method provided in an embodiment of the present application;

[0116] Figure 7 is a flow chart of an image retrieval method provided in an embodiment of the present application;

[0117] Figure 8 is a schematic diagram of an image feature library provided in an embodiment of the present application;

[0118] Fig. 9 is a flowchart of another image retrieval method provided by an embodiment of the present application;

[0119] Fig.10 It is a structural schematic diagram of a training device for a feature extraction model provided in an embodiment of the present application;

[0120] Fig.11 It is a structural schematic diagram of another feature extraction model training device provided in an embodiment of the present application;

[0121] Fig.12 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0122] Fig.13 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0123] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0124] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, a first image may be referred to as a second image, and similarly, a second image may be referred to as a first image without departing from the scope of this application.

[0125] Among them, at least one refers to one or more than one, for example, at least one image can be one image, two images, three images, or any other integer greater than or equal to one. Multiple refers to two or more than two, for example, multiple images can be two images, three images, or any other integer greater than or equal to two. Each refers to each of at least one, for example, each image refers to each image in the multiple images, and if the multiple images are 3 images, each image refers to each image in the 3 images.

[0126] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0127] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0128] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0129] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D (3-Dimension) technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0130] The following will explain the training method of the feature extraction model provided in the embodiment of the present application based on artificial intelligence technology and computer vision technology.

[0131] The training method of the feature extraction model provided in the embodiment of the present application is performed by a computer device. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms. Server. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, a car device, etc., but is not limited to this.

[0132] In one possible implementation, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. Multiple computer devices distributed at multiple locations and interconnected through a communication network can constitute a blockchain system.

[0133] In one possible implementation, the computer device used to train the feature extraction model in the embodiment of the present application is a node in the blockchain system, and the node can store the trained feature extraction model in the blockchain. Thereafter, the node or the node corresponding to other devices in the blockchain can extract image features of any image based on the feature extraction model.

[0134] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application, see Figure 1 , the implementation environment includes: a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network. Optionally, the server 102 is used to train a feature extraction model using the method provided in the embodiment of the present application, and the feature extraction model is used to extract image features of any image. The server 102 sends the trained feature extraction model to the terminal 101, and the terminal 101 can call the feature extraction model to extract image features of any image, and the image features can be applied to various fields such as image recognition or image retrieval.

[0135] In a possible implementation, an application client provided by the server is run in the terminal 101, and the server 102 stores the trained feature extraction model in the application client, and the application client has an image processing function. The terminal 101 calls the feature extraction model based on the application client to extract image features of any image.

[0136] It should be noted that Figure 1In the description, only the example of the server 102 training the feature extraction model and sending it to the terminal 101 is used for explanation. In another embodiment, the terminal 101 may also directly train the feature extraction model.

[0137] The feature extraction model training method provided in the embodiment of the present application can be applied to any scenario where image features need to be extracted.

[0138] For example, in the scenario of image retrieval, image retrieval refers to searching for images that are semantically similar to the query image from an image library. Semantic similarity can be similarity in the categories to which the images belong, such as the same objects contained in the images, or the same colors of the images. The computer device uses the method provided in the embodiment of the present application to train a feature extraction model, and then uses the feature extraction model to extract image features of each image in the image library. When it is necessary to retrieve an image similar to a certain image A, the feature extraction model is used to extract the image features of the image A, and then by comparing the image features of the image A with the image features of the images in the image library, images similar to the image A are retrieved from the image library.

[0139] For another example, in the scenario of image recognition, image recognition refers to identifying the category to which an image belongs, such as identifying whether the object contained in the image is a person, an animal, or a plant. The computer device uses the method provided in the embodiment of the present application to train a feature extraction model. When it is necessary to identify the category to which a certain image B belongs, the feature extraction model is used to extract the image features of the image B. Since the image features contain features for indicating the category to which the image belongs, the category to which the image B belongs can be determined based on the image features of the image B.

[0140] Figure 2 is a flowchart of a method for training a feature extraction model provided in an embodiment of the present application. The embodiment of the present application is executed by a computer device. Figure 2 , the method comprising:

[0141] 201. A computer device obtains a sample image and a sample category feature, where the sample category feature represents the category to which the sample image belongs.

[0142] The sample image and the sample category feature corresponding to the sample image are used to train the feature extraction model. The sample image may be an image belonging to any category. For example, if the sample image belongs to a person image, the sample category feature indicates that the sample image belongs to a person image. The sample image may be an image obtained by any means. For example, the sample image is an image captured from a video, or an image captured by a camera, or an image synthesized by a computer device.

[0143] 202. The computer device calls the feature extraction model to extract features from the sample image to obtain image features.

[0144] The feature extraction model is used to extract image features of an image. After the computer device obtains a sample image, the sample image is input into the feature extraction model, and the feature extraction model extracts features from the sample image and outputs the image features of the sample image.

[0145] 203. The computer device divides the image features into semantic features and non-semantic features.

[0146] The semantic feature represents the semantics of the sample image predicted by the feature extraction model, and the non-semantic feature is other features of the image features except the semantic feature. For example, the non-semantic feature represents the texture, brightness or color of the sample image predicted by the feature extraction model, which is irrelevant to semantics.

[0147] Among them, the image features extracted by the feature extraction model are the features in the entire feature space of the sample image. The embodiment of the present application divides the image features into semantic features and non-semantic features, which is equivalent to dividing the entire feature space into two subspaces, one is the semantic subspace and the other is the non-semantic subspace. The semantic subspace and the non-semantic subspace are complements of each other. The semantic features are the features in the semantic subspace, and the non-semantic features are the features in the non-semantic subspace.

[0148] 204. The computer device trains the feature extraction model based on the first difference between the semantic feature and the sample category feature, so as to reduce the difference between the semantic feature divided by the trained feature extraction model and the sample category feature.

[0149] In the embodiment of the present application, the training goal of the feature extraction model is to enable the semantic features obtained by dividing the extracted image features to represent the category to which the image belongs. Since the sample category feature can represent the actual category to which the sample image belongs, the closer the semantic feature obtained by dividing the image feature is to the sample category feature, the more accurate the semantic feature is, and the more accurate the image feature is, that is, the higher the feature extraction model's ability to extract features is. Therefore, the computer device determines the first difference between the semantic feature and the sample category feature, and based on the first difference, trains the feature extraction model to reduce the difference between the semantic feature divided by the trained feature extraction model and the sample category feature.

[0150] Since the difference between the semantic features divided by the trained feature extraction model and the sample category features is reduced, the semantic features divided by the trained feature extraction model are used to represent the category-related semantics in the image. For example, if the feature extraction model is trained using sample category images belonging to animal images, the semantic features are used to represent the animal-related features in the image, while features in the image that are not related to animals, such as human features and plant features, are all non-semantic features.

[0151] In addition, if the image features are not divided and the feature extraction model is trained directly based on the difference between the image features and the sample category features, the feature extraction model will only learn the semantic features related to the category in the image, while losing the non-semantic features in the image that are not related to the category.

[0152] It should be noted that, in the embodiment of the present application, steps 201 to 204 are only described by taking one sample image and the corresponding sample category feature to train the feature extraction model once as an example. In actual applications, in order to train the feature extraction model, the computer device first obtains multiple sample images and corresponding sample category features as a sample data set, and the process of training the feature extraction model based on the sample data set includes multiple iterations, and training is performed based on at least one sample image and the corresponding sample category feature in each iteration.

[0153] The method provided in the embodiment of the present application divides the image features extracted by the feature extraction model into semantic features and non-semantic features, wherein the semantic features represent the features in the image that are related to semantics, and the non-semantic features represent the features in the image that are not related to semantics, and the feature extraction model is trained by the difference between the semantic features of the sample image and the corresponding sample category features, so that the feature extraction model can not only learn the semantic features related to the category in the image, but also will not lose the non-semantic features in the image that are not related to the category. Therefore, the image features extracted by the feature extraction model trained by this method can fully characterize the features of the image, thereby improving the accuracy of the image features extracted by the feature extraction model.

[0154] Above Figure 2 The embodiment briefly describes the process of training the feature extraction model. Figure 2 The embodiments are described in detail.

[0155] First, the description of the above step 201 is as follows:

[0156] In one possible implementation, the process of a computer device determining a sample category feature corresponding to the sample image includes: the computer device generates multiple category features, each category feature includes multiple numerical values, and the number of each type of numerical values ​​in the multiple numerical values ​​is equal; the multiple category features are respectively assigned to multiple preset categories, and each preset category corresponds to a different category feature; multiple categories to which the sample image belongs are determined; and the category features corresponding to the multiple categories constitute the sample category feature.

[0157] The computer device assigns the multiple category features to multiple preset categories, which is equivalent to assigning each preset category with a corresponding category feature, and the category feature corresponding to each preset category is used to represent the preset category. Therefore, the category features corresponding to the multiple categories to which the sample image belongs are used to represent the multiple categories, and the category features corresponding to the multiple categories constitute the sample category features, which can be used to represent the multiple categories to which the sample image belongs.

[0158] Among them, since each category feature includes multiple values, and the number of each value in the multiple values ​​is equal, the various values ​​in each category feature are evenly distributed. Compared with the features obtained by using the one-hot encoding method, the sample category features obtained by this method are evenly distributed. Therefore, each position in the sample category feature is evenly activated, so that each position can carry richer semantic information, and a rich multi-category representation is achieved in semantic representation, which can avoid the problem that the one-hot encoding method cannot effectively utilize multiple positions in the feature. Therefore, the subsequent use of the sample category feature as the training target of the semantic feature divided by the feature extraction model can enable the feature extraction model to further explore more semantic features in each semantic representation dimension of the image, which is conducive to improving the learning ability of the feature extraction model in the semantic space of the image.

[0159] Optionally, the multiple category features are category feature vectors, and the computer device converts the sample category features formed by the category features corresponding to the multiple categories into a category feature matrix.

[0160] Optionally, the computer device obtains a multi-category label of the sample image, and the multi-category label can be obtained by manual annotation, or a multi-label prediction model can be used to predict the multi-category label of the sample image. The computer device determines the multiple categories to which the sample image belongs based on the multi-category label of the sample image. For example, the multi-category label can be a category identifier corresponding to the multiple categories.

[0161] Optionally, the process of generating multiple category features by a computer device includes: the computer device generates a Hadamard matrix, the Hadamard matrix includes multiple Hadamard vectors; and determining other Hadamard vectors in the Hadamard matrix except the Hadamard vectors in the first row as category features.

[0162] The computer device can generate a corresponding Hadamard matrix according to the number of preset categories. For example, if the number of preset categories is a, the computer device generates a b-order Hadamard matrix, which includes b Hadamard vectors. However, the first row of Hadamard vectors cannot be used as category features. Therefore, when generating the Hadamard matrix, it is only necessary to ensure that b is greater than a, where a and b are both positive integers. Then, the computer device randomly selects a Hadamard vectors from other Hadamard vectors except the first row of Hadamard vectors, and randomly assigns the a preset categories.

[0163] Optionally, the computer device determines the order of the Hadamard matrix according to the number of bits of the image features extracted in the following step 202. For example, the number of bits of the image features is Nh, where the number of bits refers to the number of positions in the image features, and the order of the Hadamard matrix is ​​n. In order to ensure that the image features can be subsequently divided into semantic features and non-semantic features, and that the semantic features are close to the sample category features formed by multiple Hadamard vectors through training, it is necessary to ensure that the number of bits of the image features Nh is greater than the order n of the Hadamard matrix, for example, Nh=n*P, and P is an integer greater than 1, and Nh and n are both positive integers.

[0164] The Hadamard matrix is ​​a symmetric matrix, and the trace of the Hadamard matrix is ​​0. An n-order Hadamard matrix is ​​a square matrix consisting of n*n values. Among them, the n-order Hadamard matrix can be regarded as composed of n row vectors, and the n row vectors can be called Hadamard vectors, and the n Hadamard vectors are different from each other. In addition, in the Hadamard matrix, the first row of Hadamard vectors is a vector of all 1s. Except for the first row of Hadamard vectors, the other Hadamard vectors are composed of 1 and -1, and the number of 1 and -1 is equal. Since the number of 1 and -1 in the Hadamard vector is the same, the position of 1 can be used as an activation bit. Unlike the one-hot (unique hot encoding) vector, which has only one activation bit, half of the Hadamard vector is an activation bit and the other half is an inactive bit, so it is evenly activated. Compared with the one-hot vector, the Hadamard vector can achieve richer feature expression.

[0165] The Hadamard matrix is ​​generated as follows: for an n-order Hadamard matrix H, its n+1-order Hadamard matrix is:

[0166]

[0167] For example, the Hadamard matrix of order 4 is as follows:

[0168]

[0169] Among them, the first row of Hadamard vectors in the 4th-order Hadamard matrix is ​​all 1, and the other Hadamard vectors are composed of 1 and -1, and the number of 1 and -1 is equal.

[0170] Next, the description of the above step 202 is as follows:

[0171] In one possible implementation, the feature extraction model is an Imagenet (a general object recognition open source dataset) pre-trained model, and the computer device trains a deep learning network model based on Imagenet to obtain the Imagenet pre-trained model.

[0172] In another possible implementation, the feature extraction model includes a convolutional network and a hash network. The convolutional network is used to extract features of the image, and the hash network is used to map the features extracted by the convolutional network to a hash space to obtain hash features, which are the image features extracted by the feature extraction model.

[0173] Optionally, the convolutional network can be a ResNet-101 network (a network for extracting features) or a resnet18CNN network (another network for extracting features). Taking the convolutional network as a ResNet-101 network as an example, the structure of the ResNet-101 network is shown in Table 1 below.

[0174] Table 1

[0175]

[0176] As shown in Table 1 above, ResNet-101 includes 5 convolutional layers and 1 pooling layer. In the first convolutional layer, 64 convolution kernels with a step size of 2 and a size of 7*7 are used; in the second convolutional layer, a maximum pooling function with a step size of 2 and a size of 3*3 is used, and 3 residual blocks are used. The 3 residual blocks use convolution kernels of sizes 1*1 and 3*3; 4 residual blocks are used in the third convolutional layer; 23 residual blocks are used in the fourth convolutional layer; 3 residual blocks are used in the fifth convolutional layer, and ResNet-101 uses a total of 34 residual blocks; a maximum pooling function is used in the pooling layer.

[0177] Optionally, the hash network may be composed of a fully connected function, or composed of a plurality of cascaded functions. Taking the hash network composed of a fully connected function as an example, the structure of the hash network is shown in Table 2 below.

[0178] Table 2

[0179] Network layer name Output size Layer Hashing Layer 1*256 Fully connected

[0180] As shown in Table 1 above, the hash network includes a hash layer, which is composed of a fully connected function, and the output size of the hash layer is 1*256.

[0181] Next, the description of the above step 203 is as follows:

[0182] In a possible implementation, a computer device obtains a semantic mask and a non-semantic mask, wherein the semantic mask indicates the position of a semantic feature of any image in an image feature, and the non-semantic mask indicates the position of a non-semantic feature of any image in an image feature. The computer device performs mask processing on the image feature based on the semantic mask, and determines the obtained feature as the semantic feature; and performs mask processing on the image feature based on the non-semantic mask, and determines the obtained feature as the non-semantic feature.

[0183] The semantic mask and the non-semantic mask are preset by the computer device. For example, the researchers can customize the semantic mask and the non-semantic mask through the computer device to flexibly set the positions of the semantic features and the non-semantic features in the image features.

[0184] Optionally, the image feature is a 1*Nh vector, the semantic mask Mc is a 1*Nh vector composed of 0 and 1, the position of 1 in the semantic mask indicates the position of the semantic feature in the image feature, and the position of 0 in the semantic mask indicates the position of the non-semantic feature in the image feature. Then the non-semantic mask is 1-Mc, which is also a 1*Nh vector composed of 0 and 1, the position of 1 in the non-semantic mask indicates the position of the non-semantic feature in the image feature, and the position of 0 in the semantic mask indicates the position of the semantic feature in the image feature. For example, Nh=4, semantic mask Mc=[0, 1, 0, 1], which means that the 2nd and 4th bits in the image feature are semantic features, and the 1st and 3rd bits are non-semantic features.

[0185] In another possible implementation, the computer device obtains a mapping relationship, which refers to a mapping relationship between a position in an image feature of any image and a position in any category feature; based on the mapping relationship, multiple second positions in the image feature that have a mapping relationship with multiple first positions in the sample category feature are determined. The features at the multiple second positions in the image feature constitute semantic features, and the features other than the features at the multiple second positions in the image feature constitute non-semantic features.

[0186] For example, the image feature is a vector of 1*Nh, and the category feature is a vector of 1*Nm. The number of digits Nh of the image feature is greater than the number of digits Nm of the category feature. There is a mapping relationship between each position in the category feature and a position in the image feature. The mapping relationship between the position in the category feature and the position in the image feature means that the feature at the position in the category feature and the feature at the position in the image feature belong to the same dimension. Therefore, the feature at the second position in the image feature that has a mapping relationship with the first position of the sample category feature can constitute a semantic feature, which is subsequently compared with the sample category feature, while the feature at the position in the image feature that does not have a mapping relationship with the first position of the sample category feature is a non-semantic feature other than the semantic feature.

[0187] Figure 3 is a schematic diagram of a mapping relationship provided in an embodiment of the present application, such as Figure 3 As shown, the square represents the sample category feature, the circle represents the image feature, there is a mapping relationship between the first bit in the sample category feature and the first bit in the image feature, there is a mapping relationship between the second bit in the sample category feature and the third bit in the image feature, and there is a mapping relationship between the third bit in the sample category feature and the fifth bit in the image feature. For the convenience of calculation, a mapping formula can be used to express the mapping relationship between various positions. The mapping formula is s=P*i+q, where P represents the multiple of the number of bits of the image feature and the number of bits of the sample category feature, i represents the i-th bit in the sample category feature, i is a positive integer, q represents the offset, and s represents the s-th bit in the image feature. For Figure 3 For the mapping relationship in, P=2, q=-1.

[0188] Finally, the description of the above step 204 is as follows:

[0189] In one possible implementation, the computer device determines a first loss parameter based on a first difference between the semantic feature and the sample category feature, and trains the feature extraction model based on the first loss parameter.

[0190] The computer device determines a first difference between the semantic feature and the sample category feature, and determines a first loss parameter based on the first difference, wherein the first loss parameter is positively correlated with the first difference. The computer device trains the feature extraction model based on the first loss parameter so that the first loss parameter obtained by the trained feature extraction model is reduced.

[0191] In the embodiment of the present application, since the image features include semantic features and non-semantic features, the image features will not be restricted by the category to which the sample image belongs. The image features can effectively characterize the entire feature space and fully learn the semantic subspace, thereby improving the accuracy of the image features extracted by the feature extraction model. In addition, a non-semantic subspace is also divided from the entire feature space, so in addition to the features that characterize the category in the sample image, the non-semantic features in the non-semantic subspace are also included, so it is possible to avoid the image features being overly concentrated on the features related to the category to which the sample image belongs due to the limitations of the sample image, thereby avoiding the problem of insufficient perception of the entire feature space.

[0192] In a possible implementation, the sample image includes a first sample image and a second sample image, the first sample image is similar to the second sample image, and the first sample image and the second sample image respectively correspond to their own sample category features and image features.

[0193] The computer device uses the following formula to determine the first loss parameter.

[0194] L reg =(Y a -Hada a ) 2 +(Y p -Hada p ) 2 ;

[0195] Among them, L reg represents the first loss parameter, Y a -Hada a represents the first difference between the semantic features of the first sample image and the sample category features, Y p -Hada p Y represents the first difference between the semantic feature of the second sample image and the sample category feature. a Represents the semantic features of the first sample image, Hada a represents the sample category feature of the first sample image, Y p Represents the semantic features of the second sample image, Hada p Represents the sample category feature of the second sample image.

[0196] Optionally, the semantic feature can be determined in the image feature according to the above mapping formula. Then the computer device uses the following formula to determine the first loss parameter.

[0197]

[0198] Among them, L regrepresents the first loss parameter, a represents the first sample image, p represents the second sample image, j represents the sample image, n represents the number of bits of the sample category feature, and i is a positive integer. Represents the Pi+qth feature in the image features of the sample image, Represents the i-th feature in the sample category feature of the sample image. P represents the multiple of the number of bits of the image feature and the number of bits of the sample category feature, and q represents the offset.

[0199] The method provided in the embodiment of the present application divides the image features extracted by the feature extraction model into semantic features and non-semantic features, wherein the semantic features represent the features in the image that are related to semantics, and the non-semantic features represent the features in the image that are not related to semantics, and the feature extraction model is trained by the difference between the semantic features of the sample image and the corresponding sample category features, so that the feature extraction model can not only learn the semantic features related to the category in the image, but also will not lose the non-semantic features in the image that are not related to the category. Therefore, the image features extracted by the feature extraction model trained by this method can fully characterize the features of the image, thereby improving the accuracy of the image features extracted by the feature extraction model.

[0200] Moreover, since each category feature includes multiple values, and the number of each type of values ​​in the multiple values ​​is equal, the various values ​​in each category feature are evenly distributed. Compared with the features obtained by using the one-hot encoding method, the sample category features obtained by this method are evenly distributed. Using the sample category features as the training targets of the semantic features divided by the feature extraction model can enable the feature extraction model to further mine more semantic features in each semantic representation dimension of the image, which is conducive to improving the learning ability of the feature extraction model in the semantic space of the image.

[0201] The above embodiment only takes the training of the feature extraction model based on the first difference between the semantic feature and the sample category feature as an example. In addition, the computer device can also use other methods to obtain other types of information and train the feature extraction model in combination with other types of information. Therefore, the embodiment of the present application also provides the following three methods for training the image detection model.

[0202] The first training method: the computer device trains the feature extraction model based on the first difference and the uncertainty parameter of the non-semantic feature, so that the difference between the semantic feature divided by the trained feature extraction model and the sample category feature is reduced, and the uncertainty parameter of the divided non-semantic feature is increased.

[0203] Among them, the uncertainty parameter represents the degree of uncertainty of the non-semantic feature. The higher the degree of uncertainty of the non-semantic feature, the more chaotic the non-semantic feature is, the more different non-semantic features there are, and the more features that the non-semantic feature can represent are becoming more and more abundant. In other words, the lower the degree of uncertainty of the non-semantic feature, the greater the possibility that the non-semantic features corresponding to different images are the same, resulting in the non-semantic features of different images being difficult to fully distinguish the features of different images. The higher the degree of uncertainty of the non-semantic feature, the greater the possibility that the non-semantic features corresponding to different images are different, so the non-semantic feature can fully represent the features of different images. Therefore, the embodiment of the present application trains the feature extraction model by maximizing the degree of uncertainty of the non-semantic feature, so that the feature extraction model can extract rich non-semantic features.

[0204] In one possible implementation, a computer device determines a first loss parameter, which is positively correlated with the first difference, determines a second loss parameter, which is negatively correlated with the uncertainty parameter, and trains the feature extraction model based on the first loss parameter and the second loss parameter.

[0205] Among them, the first loss parameter is the same as the first loss parameter in the above embodiment, and will not be repeated here. Since the uncertainty parameter of the non-semantic feature needs to be maximized, the second loss parameter corresponding to the non-semantic feature is negatively correlated with the uncertainty parameter, that is, the larger the uncertainty parameter, the smaller the second loss parameter, and the smaller the uncertainty parameter, the larger the second loss parameter. The computer device trains the feature extraction model based on the first loss parameter and the second loss parameter to reduce the first loss parameter and the second loss parameter obtained by the trained feature extraction model.

[0206] Optionally, the uncertainty parameter of the non-semantic feature is the information entropy of the non-semantic feature. Wherein, the information entropy is used to describe the degree of uncertainty of the information source. Optionally, the computer device uses the following formula to determine the second loss parameter.

[0207]

[0208] Among them, L ehtropy represents the second loss parameter, The information entropy of non-semantic features is also the uncertainty parameter of non-semantic features. P, j and n are positive integers, P represents the multiple between the number of bits of image features and the number of bits of semantic features, n represents the number of bits of semantic features, and the number of bits of image features is Nh, so the number of bits of non-semantic features is Nh-n=n*Pn=(P-1)*n, p j Represents the j-th feature in the non-semantic feature.

[0209] In an embodiment of the present application, in order to satisfy the uniform distribution of the non-semantic subspace of the feature space, the feature extraction model is trained by maximizing the degree of uncertainty of the non-semantic subfeatures in the non-semantic subspace, so that the non-semantic subfeatures in the image features are sufficiently dispersed to support the representation requirements of different images, thereby mining more semantically irrelevant features in the image, and thus avoiding the extracted image features from being overly concentrated on features of certain dimensions in the image, which is beneficial to improving the expression effect of the image features in the entire feature space. The subsequent use of the image features extracted by the feature extraction model for image retrieval is also beneficial to improving the accuracy of image retrieval.

[0210] The second training method: the computer device performs binary quantization on the image feature to obtain a quantized feature; based on the first difference and the second difference between the image feature and the quantized feature, the feature extraction model is trained to reduce the difference between the semantic feature divided by the trained feature extraction model and the sample category feature, and the difference between the image feature obtained by the trained feature extraction model and the corresponding quantized feature is reduced.

[0211] In order to facilitate the subsequent processing of image features extracted by the feature extraction model, the image features can be constrained into a quantitative feature that has been binary quantized. Therefore, the embodiment of the present application trains the feature extraction model by minimizing the difference between the image features and the corresponding quantitative features, so that the feature extraction model can extract image features that are close to the quantitative features.

[0212] Binary quantization refers to quantizing the values ​​in image features to two specified values, such as quantizing the values ​​in image features to 0 and 1, or quantizing the values ​​in image features to -1 and 1, etc. Taking the quantization of the values ​​in image features to -1 and 1 as an example, the training goal of the feature extraction model is to make the values ​​in the image features close to 1 or -1. Among them, the computer device can use the sign function to perform binary quantization on the image features to obtain quantized features. The process is as shown in the following formula:

[0213]

[0214] Among them, B i represents the i-th feature after quantization, i is a positive integer, Qi represents the i-th feature before quantization, sign(·) represents the sign function, when Qi is greater than 0, Qi is quantized to 1, when Qi is not greater than 0, Qi is quantized to -1.

[0215] In one possible implementation, the computer device determines a first loss parameter, which is positively correlated with the first difference; determines a third loss parameter, which is positively correlated with the second difference; and trains the feature extraction model based on the first loss parameter and the third loss parameter.

[0216] Among them, the first loss parameter is the same as the first loss parameter in the above embodiment, and will not be repeated here. Since it is necessary to minimize the second difference between the image feature and the corresponding quantitative feature, the third loss parameter corresponding to the non-semantic feature is positively correlated with the second difference, that is, the larger the second difference, the larger the third loss parameter, and the smaller the second difference, the smaller the third loss parameter. The computer device trains the feature extraction model based on the first loss parameter and the third loss parameter to reduce the first loss parameter and the third loss parameter obtained by the trained feature extraction model.

[0217] Optionally, the computer device uses the following formula to determine the third loss parameter.

[0218]

[0219] Among them, Qi represents the i-th feature before quantization, that is, the i-th feature in the image feature, B i It represents the i-th feature after quantization, that is, the i-th feature in the quantized feature, i is a positive integer, and the number of bits of the image feature is 256.

[0220] The third training method: the computer device trains the feature extraction model based on at least one first difference, a third difference and a fourth difference, so that the difference between the semantic features divided by the trained feature extraction model and the sample category features is reduced, and the difference between the image features of dissimilar images obtained by the trained feature extraction model is greater than the difference between the image features of similar images.

[0221] The sample image includes a first sample image, a second sample image and a third sample image, the second sample image is similar to the first sample image, and the third sample image is not similar to the first sample image. The at least one first difference is the difference between the semantic feature and the sample category feature of at least one of the first sample image, the second sample image and the third sample image, the third difference is the difference between the image feature of the first sample image and the image feature of the second sample image, and the fourth difference is the difference between the image feature of the first sample image and the image feature of the third sample image.

[0222] In the embodiment of the present application, the image features of the sample image represent the features in the entire feature space of the sample image. For the similar first sample image and the second sample image, the smaller the third difference between the image features of the first sample image and the image features of the second sample image, the more accurate the image features extracted by the feature extraction model are. For the dissimilar first sample image and the third sample image, the larger the fourth difference between the image features of the first sample image and the image features of the third sample image, the more accurate the image features extracted by the feature extraction model are. Therefore, it can be considered that the greater the difference between the image features of dissimilar images is than the difference between the image features of similar images, the more accurate the image features extracted by the feature extraction model are. Therefore, the computer device trains the feature extraction model with the difference between the image features of dissimilar images being greater than the difference between the image features of similar images as the training target, so that the image features extracted by the feature extraction model are more accurate.

[0223] In one possible implementation, a computer device determines a first loss parameter, which is positively correlated with the at least one first difference; determines a fourth loss parameter, which is negatively correlated with a fifth difference, and the fifth difference is the difference between the third difference and the fourth difference; and trains the feature extraction model based on the first loss parameter and the fourth loss parameter.

[0224] Among them, the first loss parameter is the same as the first loss parameter in the above embodiment, and will not be repeated here. Since the training target is that the difference between the image features of dissimilar images is greater than the difference between the image features of similar images, the fourth loss parameter corresponding to the image feature is negatively correlated with the fifth difference, that is, the larger the fifth difference, the smaller the fourth loss parameter, and the smaller the fifth difference, the larger the fourth loss parameter. The computer device trains the feature extraction model based on the first loss parameter and the fourth loss parameter to reduce the first loss parameter and the fourth loss parameter obtained by the trained feature extraction model.

[0225] Optionally, the computer device sets a target difference, and the training goal is to make the fifth difference greater than the target difference, for example, the target difference is 160, etc. The computer device may use the following formula to determine the fourth loss parameter.

[0226] L triplet =max(||Q a -Q p ||-||Q a -Q n ||+α, 0);

[0227] Among them, L triplet represents the fourth loss parameter, Q a represents the image features of the first sample image, Qp represents the image features of the third sample image, Q n represents the image features of the second sample image, α represents the target difference, ||Q a -Q p ||-||Q a -Q n || represents the fifth difference, ||Q a -Q p || represents the fourth difference, ||Q a -Q n || indicates the third difference. max(·) indicates taking the maximum value.

[0228] In an embodiment of the present application, a feature extraction model is trained by comparing the differences between image features of similar images and the differences between image features of dissimilar images, so that the image features extracted by the feature extraction model can distinguish between similar images and dissimilar images, thereby realizing metric learning of image features and improving the accuracy of image features in the entire image feature space.

[0229] In one possible implementation, the computer device may combine the second training method and the third training method to learn from the two perspectives of quantification and measurement to improve the accuracy of image features. For example, the computer device performs weighted fusion on the third loss parameter and the fourth loss parameter to obtain the fifth loss parameter, and trains the feature extraction model based on the first loss parameter and the fifth loss parameter. The weight of the third loss parameter can be set to 0.5, and the weight of the fourth loss parameter can be set to 1. Since the third loss parameter converges faster than the fourth loss parameter, in order to ensure that the fourth loss parameter is dominant in the training process, thereby ensuring that the image features have the ability to measure similarity, the weight of the fourth loss parameter is set to be greater than the weight of the third loss parameter.

[0230] It should be noted that the above embodiments respectively provide three training methods, and in another embodiment, the above three training methods can be combined arbitrarily, that is, the computer device can train the feature extraction model based on the first loss parameter, and at least one of the second loss parameter, the third loss parameter or the fourth loss parameter.

[0231] In one possible implementation, a computer device performs weighted fusion on a first loss parameter, a second loss parameter, a third loss parameter, and a fourth loss parameter to obtain a target loss parameter, and trains a feature extraction model based on the target loss parameter.

[0232] Optionally, the computer device uses m triplets to train the feature extraction model, each triplet includes three sample images, and each triplet includes sample image A and sample image B, sample image A and sample image B are similar images, and the i-th triplet also includes a sample image Ci that is not similar to sample image a, m is a positive integer, and i is a positive integer not greater than m. Then the computer device uses the following formula to determine the target loss parameter.

[0233]

[0234] Among them, L total represents the target loss parameter, w1, w2 and w3 represent weights, L hash [i] represents the fifth loss parameter corresponding to the i-th triplet, which is obtained by weighted fusion of the third loss parameter and the fourth loss parameter. represents the sum of the first loss parameters corresponding to sample image A and sample image B, Represents the sum of the second loss parameters corresponding to sample image A and sample image B.

[0235] Optionally, w1 = 0.7, w2 = 0.1, and w3 = 0.3. The reason for setting w2 to 0.1 is that the convergence speed of metric learning is slow, and it is more difficult to find similar and dissimilar parts in two images than to find similar parts in multiple images of the same category. Therefore, w1 is set to a larger value and w2 is set to a smaller value to accelerate the convergence speed of metric learning and reduce the convergence speed of semantic learning.

[0236] In an embodiment of the present application, a plurality of loss parameters determined in different ways are provided to train the feature extraction model from different angles, so that the feature extraction model can learn the deep features in a variety of images, which is beneficial to improving the robustness and accuracy of the feature extraction model.

[0237] Figure 4 is a schematic diagram of a method for training a feature extraction model provided in an embodiment of the present application, such as Figure 4As shown, the computer device randomly selects n Hadamard vectors from the Hadamard matrix, and assigns the n Hadamard vectors to different preset categories, where n is a positive integer and is less than the order of the Hadamard matrix. The computer device forms the sample category feature with the category feature corresponding to the category to which the sample image belongs. The computer device extracts features of the sample image through the convolution network and the hash network in the feature extraction model to obtain the image features of the sample image, and divides the image features into semantic features and non-semantic features according to the semantic mask and the non-semantic mask, and then determines the loss parameters of the semantic features based on the semantic features and the sample category features, determines the loss parameters of the non-semantic features based on the non-semantic features, and determines the loss parameters of the image features based on the image features, and then trains the feature extraction model using these three loss parameters.

[0238] Among them, the training process of the feature extraction model involves the following:

[0239] (1) Parameter initialization: In the pre-training stage, the network layers shown in Tables 1 and 2 are pre-trained on the ImageNet dataset. The pooling layer is initialized with a Gaussian distribution with a variance of 0.01 and a mean of 0. The hashing layer is initialized with a random normal distribution with a variance of 0.01 and a mean of 0.

[0240] (2) Learning rate: The feature extraction model is trained with a learning rate of 0.0005, and the learning rate is increased by 0.1 times every 10 rounds of iterative training. Multiple rounds of iterations are performed based on different sample images until the loss parameter no longer decreases.

[0241] (3) The operations in each round of iteration are as follows: mining the above triplets in multiple sample images, and performing model forward calculation, loss calculation and model parameter update based on the mined triplets.

[0242] The forward calculation of the model includes: first, setting the parameters of the feature extraction model to the learning state. During the training process, the feature extraction model performs forward calculation on the input sample image to obtain the image feature, which can be represented by Q, where Q is a 1x256 vector. Then, the image feature is subjected to semantic masking and non-semantic masking to obtain semantic features and non-semantic features. For example, Q = [0.1, 0.2, 0.7, -0.5], semantic mask Mc = [0, 1, 0, 1], non-semantic mask Mc = [1, 0, 1, 0], then the semantic feature is [0.2, -0.5], and the non-semantic feature is [0.1, 0.7].

[0243] The loss calculation includes: for each image in each triplet, the quantization loss and the metric loss are calculated, which are the third loss parameter and the fourth loss parameter mentioned above; for similar images in each triplet, the semantic loss is calculated using the sample category features, which is the first loss parameter mentioned above; for similar images in each triplet, the non-semantic loss is calculated using the sample category features, which is the third loss parameter mentioned above. Then the three loss parameters are weighted and fused to obtain the target loss parameter.

[0244] Among them, the model parameter update includes: using SGD (Stochastic Gradient Descent) to perform gradient backward calculation on the target loss parameter, obtain the updated value of each model parameter in the feature extraction model, and update the feature extraction model.

[0245] In the related art, the feature extraction model is trained by metric learning. Since metric learning only considers the similarity between images, the image features extracted by the feature extraction model are constrained by the training samples, and it is difficult to perceive the features in the training samples that are not related to the similarity, resulting in insufficient extracted image features. For example, a set of images of all dogs is used as a training sample. Since the dog noses are all protruding, features for distinguishing whether the dog nose protrudes outward and the degree of protrusion will not be learned. However, when it is necessary to distinguish between a cat nose and a dog nose, since the extracted image features do not fully express the dog nose, it is impossible to distinguish the feature differences between the cat nose and the dog nose. In addition, the image features learned in this way have different degrees of redundant bits, resulting in uneven distribution of image features. For example, an important feature may be represented by different numbers of bits such as 0, 1, and 2, resulting in insufficient accuracy of the image features.

[0246] In the embodiment of the present application, the feature space of the image is divided into a semantic subspace and a non-semantic subspace, and the image features include semantic features and non-semantic features, thereby fully representing the features of the entire image and avoiding the problem of insufficient overall spatial perception. In addition, the image features are learned as a whole using the similarity between images. In addition, the semantic features meet the requirements of representing the category to which the image belongs, and the non-semantic features meet the requirements of maximizing the degree of uncertainty, avoiding the problem of difficulty in fully representing due to excessive concentration of non-semantic features, thereby performing discriminative learning of semantic features and non-semantic features, and improving the representation ability of the entire image features to a certain extent.

[0247] Figure 5Schematic diagram of a feature space provided by an embodiment of the present application, feature space 501 is constructed by using the training method provided by an embodiment of the present application, and feature space 502 is constructed by using the training method of the related technology. X and Y represent sample images belonging to different categories, two underlined Xs or two underlined Ys represent two similar images, the largest circle in feature space 501 and feature space 502 represents the entire feature space, and the different small circles in feature space 501 represent the distribution of features of different categories in the feature space. Figure 5 As shown, in feature space 501, the distances between the positions of different small circles are approximately the same, so features of different categories are evenly distributed in the feature space, and features of the same category are concentrated together, and there is a certain feature space between features of the same category for representing non-semantic features that are not related to the category, which is enough to accommodate more fine-grained features. In feature space 502, the distribution of features of the same category is relatively scattered, and features of different categories are not concentrated enough, and all features are concentrated in the upper half of the feature space, so the features that can be represented by feature space 502 are not sufficient.

[0248] In the third training method, the sample images include a first sample image, a second sample image, and a third sample image. The acquisition process of the first sample image, the second sample image, and the third sample image is described in detail below. Figure 6 Embodiment of the invention.

[0249] Figure 6 is a flow chart of a sample image acquisition method provided in an embodiment of the present application. The embodiment of the present application is executed by a computer device. Figure 6 , the method comprising:

[0250] 601. A computer device obtains the first sample image, the second sample image, and a plurality of candidate sample images.

[0251] The first sample image is similar to the second sample image, and the computer device needs to determine an image that is not similar to the first sample image from the multiple candidate sample images.

[0252] In one possible implementation, a computer device obtains multiple sample pairs, each sample pair includes two similar sample images; the two sample images in a target sample pair are respectively determined as the first sample image and the second sample image, and the target sample pair is any sample pair in the multiple sample pairs; and the sample images in other sample pairs in the multiple sample pairs except the target sample pair are determined as the candidate sample images.

[0253] Among them, training the feature extraction model requires the use of similar sample pairs, each sample pair can be two similar sample images manually annotated, wherein the sample images in the sample pair can be images randomly captured from the video, or images captured by a shooting device, etc.

[0254] After obtaining the sample pairs, triplets are mined based on the sample pairs. Each triplet includes three sample images. The three sample images can be called reference sample images, positive sample images, and negative sample images. The positive sample images are similar to the reference sample images, and the negative sample images are not similar to the reference sample images. The first sample image can be called the reference sample image, and the second sample image can be called the positive sample image. The computer device needs to mine the negative sample image corresponding to the reference sample image in multiple sample pairs.

[0255] For example, in each round of iterative training, 10 triplets are mined, and the computer device randomly determines a target sample pair from multiple sample pairs, and determines the two sample images in the target sample pair as the first sample image and the second sample image respectively, and then for the other sample pairs except the target sample pair in the multiple sample pairs, any sample image in each other sample pair is determined as a candidate sample image, and then the method of the following steps 602 to 604 is used to determine 10 dissimilar images to the first sample image from the multiple candidate sample images, and each dissimilar feature is combined with the first sample image and the second sample image to form a triplet, thereby obtaining 10 different triplets.

[0256] 602. The computer device determines the similarity between the first sample image and each of the candidate sample images respectively.

[0257] In one possible implementation, a computer device calls a feature extraction model to perform feature extraction on the first sample image and each candidate sample image, respectively, to obtain a first image feature of the first sample image and a second image feature of each candidate sample image, and to determine a similarity between the first image feature and each second image feature, respectively. The similarity between the first image feature and each second image feature is the similarity between the first sample image and each of the candidate sample images.

[0258] Among them, in each iterative training, it is necessary to first mine the triplets used to train the feature extraction model. The computer device uses the feature extraction model obtained in the previous training to determine the similarity between the first sample image and each of the candidate sample images, thereby mining dissimilar images, which is equivalent to applying the training results of the previous round to the sample mining process of this round. As the number of training times of the feature extraction model increases, the accuracy of the feature extraction model also increases, and the accuracy of the sample images mined by the feature extraction model also increases. Then, using the sample images with higher accuracy, the feature extraction model is trained again, which further improves the accuracy of the feature extraction model.

[0259] Optionally, the computer device uses the distance between the first image feature and each second image feature to represent the similarity between the first image feature and each second image feature, wherein the smaller the distance between the first image feature and the second image feature, the greater the similarity between the first image feature and the second image feature, and the larger the distance between the first image feature and the second image feature, the smaller the similarity between the first image feature and the second image feature.

[0260] 603. The computer device deletes a largest first number of similarities from the determined multiple similarities.

[0261] After the computer device determines the similarity between the first sample image and each of the candidate sample images, it is considered that if the similarity between the first sample image and the candidate sample image is too large, the possibility that the first sample image and the candidate sample image are similar images is greater. However, except for intercepting continuous similar images in a video, or adding a hue change to an image to obtain a similar image, the possibility that most images are similar to each other is very small. Therefore, it can be considered that among the multiple candidate sample images, except for the first number of candidate sample images with the largest similarity, which may be images similar to the first sample image, the other candidate sample images may be images that are not similar to the first sample image. Therefore, when the computer device mines the dissimilar sample images, it first deletes the first number of similarities with the largest similarities among the multiple similarities determined.

[0262] 604. The computer device determines, among the remaining multiple similarities, candidate sample images corresponding to the largest second number of similarities as sample images that are not similar to the first sample image.

[0263] In the embodiment of the present application, it is considered that the candidate sample images corresponding to the remaining multiple similarities can all be used as sample images that are not similar to the first sample image. However, in order to improve the feature extraction capability of the feature extraction model, the learning difficulty of the feature extraction model can be increased. Therefore, among the remaining multiple similarities, the candidate sample images corresponding to the second number of similarities with relatively large similarities are determined as sample images that are not similar to the first sample image, thereby increasing the difficulty for the feature extraction model to distinguish between the first sample image and the sample image that is not similar to the first sample image.

[0264] The above embodiment illustrates the training process of the feature extraction model. After obtaining the trained feature extraction model, the feature extraction model can be applied to the image retrieval scenario. The specific process is described in detail below. Figure 7 Embodiment of the invention.

[0265] Figure 7 is a flowchart of an image retrieval method provided in an embodiment of the present application. The embodiment of the present application is executed by a computer device. Figure 7 , the method comprising:

[0266] 701. The computer device responds to a search instruction and obtains a target image carried in the search instruction.

[0267] The retrieval instruction is used to request to retrieve similar images of the target image.

[0268] 702. The computer device calls the trained feature extraction model to perform feature extraction on the target image to obtain a third image feature of the target image.

[0269] After the computer device acquires the target image, it inputs the target image into the trained feature extraction model, and the trained feature extraction model extracts features from the target image and outputs the third image features of the target image.

[0270] The process of extracting the image features of the target image in step 702 is the same as the process of extracting the image features of the sample image in step 202, and will not be described in detail here.

[0271] 703. The computer device searches for a similar image to the target image based on the third image feature.

[0272] After the computer device obtains the third image feature, it compares the third image feature with the image features of other images to determine whether the third image feature is sufficiently similar to the image features of other images. If the third image feature is sufficiently similar to the image feature of a certain image, the image is considered to be a similar image of the target image.

[0273] In a possible implementation, the image feature library includes multiple feature sets, each of which includes a central image feature and an image feature similar to the central image feature. The computer device searches for a central image feature whose similarity with the third image feature satisfies a first similarity condition in the image feature library; determines a feature set in which the found central image feature is located; searches for an image feature whose similarity with the third image feature satisfies a second similarity condition in the determined feature set, and determines an image corresponding to the found image feature as a similar image to the target image.

[0274] Among them, the image feature library includes multiple image features. By clustering the multiple image features in the image feature library, multiple feature sets can be obtained. Each feature set is a cluster cluster. Each feature set includes a central image feature and image features similar to the central image feature. The central image feature is also the cluster center in the cluster cluster. Figure 8 is a schematic diagram of an image feature library provided in an embodiment of the present application, such as Figure 8 As shown, the multiple image features 801 are image features that have not been clustered. After clustering the multiple image features 801, N clusters are obtained, each cluster is a feature set, and the multiple feature sets obtained constitute an image feature library 802.

[0275] Since all image features except the central image feature in the feature set are similar to the central image feature, the central image feature in each feature set can be used as an index when searching for image features. The computer device first determines the similarity between each central image feature and the third image feature in the image feature library, and searches for the central image feature whose similarity with the third image feature meets the first similarity condition. If the similarity between a central image feature and the third image feature meets the first similarity condition, it is considered that the central image feature is relatively similar to the third image feature, and the other image features in the feature set where the central image feature is located are also relatively similar to the third image feature. Therefore, after finding the central image feature that meets the first similarity condition, the computer device determines the feature set where the central image feature is located, and searches for the image feature whose similarity with the third image feature meets the second similarity condition in the feature set, which is equivalent to using the central image feature as an index, first searching for multiple image features that are relatively similar to the third image feature, and then searching for at least one image feature that is more similar to the central image feature among these image features, without traversing the entire image feature library, which is conducive to improving the speed of image retrieval.

[0276] Optionally, the computer device uses the distance between image features to represent the similarity between image features, and the smaller the distance between an image feature and a second image feature, the greater the similarity between the image features, and the larger the distance between the image features, the smaller the similarity between the image features. For example, the distance is the Hamming distance.

[0277] Optionally, the image features in the image feature library are features that have been binary quantized. The computer device performs binary quantization on the third image feature to obtain a target quantized feature, and in the image feature library, searches for a central image feature whose similarity with the target quantized feature satisfies the first similarity condition. In the embodiment of the present application, by first performing binary quantization on the image features and then determining the similarity between the quantized quantized features, since the quantized features only include two numerical values, the calculation method is simpler, which can accelerate the calculation efficiency and help improve the retrieval speed. Among them, the binary quantization method is the same as the process of binary quantization in the above-mentioned training process, which will not be repeated here.

[0278] Optionally, the central image feature whose similarity with the third image feature satisfies the first similarity condition refers to: the third number of central image features in the image feature library that have the greatest similarity with the third image feature. The image features whose similarity with the third image feature satisfies the second similarity condition refers to: the fourth number of image features in the determined feature set that have the greatest similarity with the third image feature. The third number and the fourth number may be any number pre-set by the computer device.

[0279] Optionally, the central image feature whose similarity with the third image feature satisfies the first similarity condition refers to: a central image feature in the image feature library whose similarity with the third image feature is greater than a first threshold. The image feature whose similarity with the third image feature satisfies the second similarity condition refers to: an image feature in the determined feature set whose similarity with the third image feature is greater than a second threshold. The first threshold and the second threshold may be any numerical values ​​pre-set by the computer device.

[0280] Fig. 9 is a flowchart of another image retrieval method provided in an embodiment of the present application. Fig. 9As shown, the computer device uses the convolutional network and hash network in the feature extraction model to extract features from multiple images in the image library to obtain image features of the multiple images, and then clusters the multiple image features to obtain multiple feature sets, which form an image feature library. After the computer device obtains the target image to be retrieved, it uses the feature extraction model to extract the third image feature of the target image, and then retrieves image features similar to the third image feature in the image feature library, and determines the images corresponding to the retrieved multiple similar image features as similar images of the target image. Fig. 9 In the figure, 8 similar images are retrieved as an example.

[0281] In the method provided in the embodiment of the present application, the feature extraction model trained in the above embodiment is used to extract image features of the target image, and then the image features are used to search for similar images of the target image. Since the image features extracted by the feature extraction model are more accurate, the accuracy of image retrieval is also improved to a certain extent.

[0282] Fig.10 Schematic diagram of a training device for a feature extraction model provided in an embodiment of the present application. Fig.10 , the device comprises:

[0283] The sample acquisition module 1001 is used to acquire a sample image and a sample category feature, where the sample category feature represents the category to which the sample image belongs;

[0284] The feature extraction module 1002 is used to call the feature extraction model to extract features from the sample image to obtain image features;

[0285] A feature division module 1003 is used to divide image features into semantic features and non-semantic features, wherein the semantic features represent the semantics of the sample image predicted by the feature extraction model, and the non-semantic features are other features of the image features except the semantic features;

[0286] The model training module 1004 is used to train the feature extraction model based on the first difference between the semantic feature and the sample category feature, so as to reduce the difference between the semantic feature and the sample category feature divided by the trained feature extraction model.

[0287] The training device of the feature extraction model provided in the embodiment of the present application divides the image features extracted by the feature extraction model into semantic features and non-semantic features, wherein the semantic features represent the features in the image that are related to semantics, and the non-semantic features represent the features in the image that are not related to semantics, and trains the feature extraction model through the difference between the semantic features of the sample image and the corresponding sample category features, so that the feature extraction model can not only learn the semantic features in the image that are related to the category, but also will not lose the non-semantic features in the image that are not related to the category. Therefore, the image features extracted by the trained feature extraction model can fully characterize the features of the image, thereby improving the accuracy of the image features extracted by the feature extraction model.

[0288] Alternatively, see Fig.11 , the feature segmentation module 1003 includes:

[0289] A mask acquisition unit 1013, used to acquire a semantic mask and a non-semantic mask, wherein the semantic mask indicates the position of the semantic feature of any image in the image feature, and the non-semantic mask indicates the position of the non-semantic feature of any image in the image feature;

[0290] A mask processing unit 1023, configured to perform mask processing on image features based on semantic masks, and determine the obtained features as semantic features;

[0291] The mask processing unit 1023 is used to perform mask processing on the image features based on the non-semantic mask, and determine the obtained features as non-semantic features.

[0292] Alternatively, see Fig.11 , the feature segmentation module 1003 includes:

[0293] A relationship acquisition unit 1033 is used to acquire a mapping relationship, where the mapping relationship refers to a mapping relationship between a position in an image feature of any image and a position in a class feature;

[0294] The position determination unit 1043 is used to determine, based on the mapping relationship, multiple second positions in the image features that have a mapping relationship with multiple first positions in the sample category features; wherein the features at the multiple second positions in the image features constitute semantic features, and the features in the image features other than the features at the multiple second positions constitute non-semantic features.

[0295] Alternatively, see Fig.11 , the sample acquisition module 1001 includes:

[0296] A generating unit 1011 is used to generate a plurality of category features, each category feature includes a plurality of values, and the number of each value in the plurality of values ​​is equal;

[0297] A classification unit 1021, configured to respectively assign a plurality of category features to a plurality of preset categories, each preset category corresponding to a different category feature;

[0298] A first determining unit 1031 is used to determine the multiple categories to which the sample image belongs;

[0299] The second determining unit 1041 is configured to form a sample category feature from category features corresponding to a plurality of categories.

[0300] Alternatively, see Fig.11 , generating unit 1011, used for:

[0301] Generate a Hadamard matrix, the Hadamard matrix includes a plurality of Hadamard vectors;

[0302] The other Hadamard vectors except the Hadamard vector of the first row in the Hadamard matrix are determined as category features.

[0303] Alternatively, see Fig.11 , the model training module 1004 includes:

[0304] A loss determination unit 1014, configured to determine a first loss parameter, wherein the first loss parameter is positively correlated with the first difference;

[0305] The training unit 1024 is used to train the feature extraction model based on the first loss parameter.

[0306] Alternatively, see Fig.11 , the model training module 1004 is used to:

[0307] Based on the first difference and the uncertainty parameter of the non-semantic feature, the feature extraction model is trained so that the difference between the semantic features and the sample category features divided by the trained feature extraction model is reduced and the uncertainty parameter of the divided non-semantic feature is increased, wherein the uncertainty parameter represents the degree of uncertainty of the non-semantic feature.

[0308] Alternatively, see Fig.11 , the model training module 1004 includes:

[0309] A loss determination unit 1014, configured to determine a first loss parameter, wherein the first loss parameter is positively correlated with the first difference;

[0310] The loss determination unit 1014 is further used to determine a second loss parameter, where the second loss parameter is negatively correlated with the uncertainty parameter;

[0311] The training unit 1024 is used to train the feature extraction model based on the first loss parameter and the second loss parameter.

[0312] Alternatively, see Fig.11, the model training module 1004 is used to:

[0313] Perform binary quantization on the image features to obtain quantized features;

[0314] Based on the first difference and the second difference between the image feature and the quantitative feature, the feature extraction model is trained so that the difference between the semantic features divided by the trained feature extraction model and the sample category features is reduced, and the difference between the image features obtained by the trained feature extraction model and the corresponding quantitative features is reduced.

[0315] Alternatively, see Fig.11 , the model training module 1004 includes:

[0316] A loss determination unit 1014, configured to determine a first loss parameter, wherein the first loss parameter is positively correlated with the first difference;

[0317] The loss determination unit 1014 is further configured to determine a third loss parameter, where the third loss parameter is positively correlated with the second difference;

[0318] The training unit 1024 is used to train the feature extraction model based on the first loss parameter and the third loss parameter.

[0319] Alternatively, see Fig.11 , the sample images include a first sample image, a second sample image and a third sample image, the second sample image is similar to the first sample image, and the third sample image is not similar to the first sample image;

[0320] The model training module 1004 is used to:

[0321] Based on at least one first difference, a third difference, and a fourth difference, training the feature extraction model so that the difference between the semantic features divided by the trained feature extraction model and the sample category features is reduced, and the difference between the image features of the dissimilar images obtained by the trained feature extraction model is greater than the difference between the image features of the similar images;

[0322] Among them, at least one first difference is the difference between the semantic features and the sample category features of at least one image among the first sample image, the second sample image and the third sample image, the third difference is the difference between the image features of the first sample image and the image features of the second sample image, and the fourth difference is the difference between the image features of the first sample image and the image features of the third sample image.

[0323] Alternatively, see Fig.11 , the model training module 1004 includes:

[0324] A loss determination unit 1014, configured to determine a first loss parameter, wherein the first loss parameter is positively correlated with at least one first difference value;

[0325] The loss determination unit 1014 is further configured to determine a fourth loss parameter, wherein the fourth loss parameter is negatively correlated with the fifth difference, and the fifth difference is the difference between the third difference and the fourth difference;

[0326] The training unit 1024 is used to train the feature extraction model based on the first loss parameter and the fourth loss parameter.

[0327] Alternatively, see Fig.11 , the sample acquisition module 1001 includes:

[0328] An image acquisition unit 1051 is used to acquire a first sample image, a second sample image, and a plurality of candidate sample images;

[0329] A similarity determination unit 1061, configured to respectively determine the similarity between the first sample image and each candidate sample image;

[0330] The image determination unit 1071 is used to delete the largest first number of similarities among the determined multiple similarities, and determine the candidate sample images corresponding to the largest second number of similarities among the remaining multiple similarities as sample images that are not similar to the first sample image.

[0331] Alternatively, see Fig.11 , the image acquisition unit 1051 is used to:

[0332] Acquire multiple sample pairs, each sample pair including two similar sample images;

[0333] Determine two sample images in a target sample pair as a first sample image and a second sample image respectively, and the target sample pair is any sample pair among the multiple sample pairs;

[0334] The sample images in the other sample pairs except the target sample pair among the multiple sample pairs are determined as candidate sample images.

[0335] Alternatively, see Fig.11 , the device further comprises:

[0336] The retrieval module 1005 is used to respond to the retrieval instruction and obtain the target image carried in the retrieval instruction;

[0337] The feature extraction module 1002 is further used to call the trained feature extraction model to extract features of the target image to obtain a third image feature of the target image;

[0338] The retrieval module 1005 is further configured to search for similar images to the target image based on the third image feature.

[0339] Alternatively, see Fig.11, the image feature library includes multiple feature sets, each feature set includes a central image feature and image features similar to the central image feature;

[0340] The search module 1005 includes:

[0341] A feature retrieval unit 1015 is used to search, in the image feature library, for a central image feature whose similarity with the third image feature satisfies a first similarity condition;

[0342] A set determining unit 1025, used to determine the feature set where the found central image feature belongs;

[0343] The image retrieval unit 1035 is used to search for an image feature whose similarity with the third image feature satisfies a second similarity condition in the determined feature set, and determine the image corresponding to the searched image feature as a similar image to the target image.

[0344] Alternatively, see Fig.11 The image features in the image feature library are binary quantized features. The feature retrieval unit 1015 is used to:

[0345] Binary quantization is performed on the third image feature to obtain a target quantized feature;

[0346] In the image feature library, a central image feature whose similarity with the target quantitative feature satisfies the first similarity condition is searched.

[0347] Alternatively, see Fig.11 , a feature retrieval unit 1015, used to: search for a third number of central image features having the greatest similarity with the third image feature in the image feature library;

[0348] The image retrieval unit 1035 is used to search for a fourth number of image features having the greatest similarity with the third image feature in the determined feature set.

[0349] It should be noted that the training device for the feature extraction model provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the training device for the feature extraction model provided in the above embodiment and the training method embodiment of the feature extraction model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0350] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the training method of the feature extraction model of the above embodiment.

[0351] Optionally, the computer device is provided as a terminal. Fig.12 A schematic diagram of the structure of a terminal 1200 provided by an exemplary embodiment of the present application is shown.

[0352] The terminal 1200 includes: a processor 1201 and a memory 1202 .

[0353] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0354] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one computer program, which is used by the processor 1201 to implement the training method of the feature extraction model provided in the method embodiment of the present application.

[0355] In some embodiments, the terminal 1200 may further optionally include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1203 via a bus, a signal line or a circuit board. Optionally, the peripheral device includes: at least one of a radio frequency circuit 1204 or a display screen 1205.

[0356] The peripheral device interface 1203 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0357] The radio frequency circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1204 can communicate with other devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1204 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0358] The display screen 1205 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 also has the ability to collect touch signals on the surface or above the surface of the display screen 1205. The touch signal may be input as a control signal to the processor 1201 for processing.

[0359] Those skilled in the art will understand that Fig.12The structure shown in the figure does not constitute a limitation on the terminal 1200, and the terminal 1200 may include more or fewer components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0360] Optionally, the computer device is provided as a server. Fig.13 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1301 and one or more memories 1302, wherein the memory 1302 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1301 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0361] An embodiment of the present application also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the training method of the feature extraction model of the above embodiment.

[0362] An embodiment of the present application also provides a computer program product, including a computer program, which is loaded and executed by a processor to implement the operations performed by the training method of the feature extraction model in the above embodiment.

[0363] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0364] The above are only optional embodiments of the embodiments of the present application and are not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A method for training a feature extraction model, characterized in that: The method comprises: Acquire a sample image and a sample category feature, wherein the sample category feature represents the category to which the sample image belongs; Calling a feature extraction model to extract features from the sample image to obtain image features; Dividing the image features into semantic features and non-semantic features, wherein the semantic features represent the semantics of the sample image predicted by the feature extraction model, and the non-semantic features are other features of the image features except the semantic features; Based on the first difference between the semantic feature and the sample category feature and the uncertainty parameter of the non-semantic feature, the feature extraction model is trained so that the difference between the semantic feature and the sample category feature divided by the trained feature extraction model is reduced and the uncertainty parameter of the divided non-semantic feature is increased, wherein the uncertainty parameter represents the degree of uncertainty of the non-semantic feature.

2. The method according to claim 1, characterized in that The dividing the image features into semantic features and non-semantic features comprises: Acquire a semantic mask and a non-semantic mask, wherein the semantic mask indicates the position of the semantic features of any image in the image features, and the non-semantic mask indicates the position of the non-semantic features of any image in the image features; Based on the semantic mask, mask processing is performed on the image feature, and the obtained feature is determined as the semantic feature; Based on the non-semantic mask, mask processing is performed on the image feature, and the obtained feature is determined as the non-semantic feature.

3. The method according to claim 1, characterized in that The dividing the image features into semantic features and non-semantic features comprises: Acquire a mapping relationship, where the mapping relationship refers to a mapping relationship between a position in an image feature of any image and a position in any category feature; Based on the mapping relationship, multiple second positions in the image feature that have a mapping relationship with multiple first positions in the sample category feature are determined, wherein the features at the multiple second positions in the image feature constitute the semantic feature, and other features in the image feature except the features at the multiple second positions constitute the non-semantic feature.

4. The method according to claim 1, characterized in that: Obtaining the sample category features includes: Generate multiple category features, each category feature includes multiple values, and the number of each value in the multiple values ​​is equal; Allocating the multiple category features to multiple preset categories respectively, each preset category corresponding to a different category feature; Determining the multiple categories to which the sample image belongs; The category features corresponding to the multiple categories constitute the sample category features.

5. The method according to claim 1, characterized in that: The step of training the feature extraction model based on the first difference between the semantic feature and the sample category feature and the uncertainty parameter of the non-semantic feature comprises: determining a first loss parameter, wherein the first loss parameter is positively correlated with the first difference; determining a second loss parameter, wherein the second loss parameter is negatively correlated with the uncertainty parameter; The feature extraction model is trained based on the first loss parameter and the second loss parameter.

6. The method according to claim 1, characterized in that The step of training the feature extraction model based on the first difference between the semantic feature and the sample category feature and the uncertainty parameter of the non-semantic feature comprises: Performing binary quantization on the image features to obtain quantized features; Based on the first difference, the uncertainty parameter and the second difference between the image feature and the quantitative feature, the feature extraction model is trained so that the difference between the semantic feature divided by the trained feature extraction model and the sample category feature is reduced, the uncertainty parameter of the divided non-semantic feature is increased, and the difference between the image feature obtained by the trained feature extraction model and the corresponding quantitative feature is reduced.

7. The method according to claim 1, characterized in that The sample images include a first sample image, a second sample image and a third sample image, the second sample image is similar to the first sample image, and the third sample image is not similar to the first sample image; The step of training the feature extraction model based on the first difference between the semantic feature and the sample category feature and the uncertainty parameter of the non-semantic feature comprises: Based on the uncertainty parameter, at least one first difference, a third difference, and a fourth difference, the feature extraction model is trained so that the difference between the semantic features divided by the trained feature extraction model and the sample category features is reduced, and the uncertainty parameter of the divided non-semantic features is increased, and the difference between the image features of dissimilar images obtained by the trained feature extraction model is greater than the difference between the image features of similar images; Among them, the at least one first difference value is the difference between the semantic features and the sample category features of at least one image among the first sample image, the second sample image and the third sample image, the third difference value is the difference between the image features of the first sample image and the image features of the second sample image, and the fourth difference value is the difference between the image features of the first sample image and the image features of the third sample image.

8. The method according to claim 7, characterized in that The obtaining of the sample image comprises: Acquire the first sample image, the second sample image, and a plurality of candidate sample images; respectively determining the similarity between the first sample image and each of the candidate sample images; Among the determined multiple similarities, deleting the largest first number of similarities; Among the remaining multiple similarities, the candidate sample images corresponding to the largest second number of similarities are determined as sample images that are not similar to the first sample image.

9. The method according to any one of claims 1 to 8, characterized in that: After training the feature extraction model based on the first difference between the semantic feature and the sample category feature and the uncertainty parameter of the non-semantic feature, the method further includes: In response to a search instruction, acquiring a target image carried in the search instruction; Calling the trained feature extraction model to perform feature extraction on the target image to obtain a third image feature of the target image; Based on the third image feature, similar images to the target image are searched.

10. The method according to claim 9, characterized in that The image feature library includes a plurality of feature sets, each of which includes a central image feature and image features similar to the central image feature; The searching for a similar image to the target image based on the third image feature includes: In the image feature library, searching for a central image feature whose similarity with the third image feature satisfies a first similarity condition; Determine the feature set where the found central image feature is located; In the determined feature set, an image feature whose similarity with the third image feature satisfies a second similarity condition is searched, and an image corresponding to the searched image feature is determined as a similar image to the target image.

11. A training device for a feature extraction model, characterized in that: The device comprises: A sample acquisition module, used to acquire a sample image and a sample category feature, wherein the sample category feature represents the category to which the sample image belongs; A feature extraction module is used to call a feature extraction model to extract features from the sample image to obtain image features; A feature division module, used for dividing the image features into semantic features and non-semantic features, wherein the semantic features represent the semantics of the sample image predicted by the feature extraction model, and the non-semantic features are other features of the image features except the semantic features; A model training module is used to train the feature extraction model based on the first difference between the semantic feature and the sample category feature and the uncertainty parameter of the non-semantic feature, so that the difference between the semantic feature and the sample category feature divided by the trained feature extraction model is reduced and the uncertainty parameter of the divided non-semantic feature is increased, wherein the uncertainty parameter represents the degree of uncertainty of the non-semantic feature.

12. The device according to claim 11, characterized in that The feature division module comprises: A mask acquisition unit, used to acquire a semantic mask and a non-semantic mask, wherein the semantic mask indicates the position of the semantic feature of any image in the image feature, and the non-semantic mask indicates the position of the non-semantic feature of any image in the image feature; A mask processing unit, configured to perform mask processing on the image feature based on the semantic mask, and determine the obtained feature as the semantic feature; The mask processing unit is used to perform mask processing on the image feature based on the non-semantic mask, and determine the obtained feature as the non-semantic feature.

13. The device according to claim 11, characterized in that The feature division module comprises: A relationship acquisition unit, used to acquire a mapping relationship, wherein the mapping relationship refers to a mapping relationship between a position in an image feature of any image and a position in a class feature; A position determination unit is used to determine, based on the mapping relationship, a plurality of second positions in the image feature that have a mapping relationship with a plurality of first positions in the sample category feature, wherein the features at the plurality of second positions in the image feature constitute the semantic feature, and the other features in the image feature except the features at the plurality of second positions constitute the non-semantic feature.

14. The device according to claim 11, characterized in that The sample acquisition module comprises: A generating unit, configured to generate a plurality of category features, each category feature comprising a plurality of values, and the number of each type of values ​​in the plurality of values ​​is equal; A classification unit, used to assign the plurality of category features to a plurality of preset categories respectively, each preset category corresponding to a different category feature; A first determining unit, configured to determine the multiple categories to which the sample image belongs; The second determining unit is configured to form the sample category feature from category features corresponding to the multiple categories.

15. The device according to claim 11, characterized in that The model training module includes: a loss determination unit, configured to determine a first loss parameter, wherein the first loss parameter is positively correlated with the first difference; The loss determination unit is further used to determine a second loss parameter, where the second loss parameter is negatively correlated with the uncertainty parameter; A training unit is used to train the feature extraction model based on the first loss parameter and the second loss parameter.

16. The device according to claim 11, characterized in that The model training module is used to: Performing binary quantization on the image features to obtain quantized features; Based on the first difference, the uncertainty parameter and the second difference between the image feature and the quantitative feature, the feature extraction model is trained so that the difference between the semantic feature divided by the trained feature extraction model and the sample category feature is reduced, the uncertainty parameter of the divided non-semantic feature is increased, and the difference between the image feature obtained by the trained feature extraction model and the corresponding quantitative feature is reduced.

17. The device according to claim 11, characterized in that The sample images include a first sample image, a second sample image and a third sample image, the second sample image is similar to the first sample image, and the third sample image is not similar to the first sample image; The model training module is used to: Based on the uncertainty parameter, at least one first difference, a third difference, and a fourth difference, the feature extraction model is trained so that the difference between the semantic features divided by the trained feature extraction model and the sample category features is reduced, and the uncertainty parameter of the divided non-semantic features is increased, and the difference between the image features of dissimilar images obtained by the trained feature extraction model is greater than the difference between the image features of similar images; Among them, the at least one first difference value is the difference between the semantic features and the sample category features of at least one image among the first sample image, the second sample image and the third sample image, the third difference value is the difference between the image features of the first sample image and the image features of the second sample image, and the fourth difference value is the difference between the image features of the first sample image and the image features of the third sample image.

18. The device according to claim 17, characterized in that The sample acquisition module comprises: An image acquisition unit, configured to acquire the first sample image, the second sample image, and a plurality of candidate sample images; a similarity determination unit, configured to respectively determine the similarity between the first sample image and each of the candidate sample images; The image determination unit is used to delete the largest first number of similarities among the determined multiple similarities; and determine the candidate sample images corresponding to the largest second number of similarities among the remaining multiple similarities as sample images that are not similar to the first sample image.

19. The device according to any one of claims 11 to 18, characterized in that: The device also includes: A retrieval module, configured to respond to a retrieval instruction and obtain a target image carried in the retrieval instruction; The feature extraction module is further used to call the trained feature extraction model to perform feature extraction on the target image to obtain a third image feature of the target image; The retrieval module is further used to search for similar images of the target image based on the third image feature.

20. The device according to claim 19, characterized in that The image feature library includes a plurality of feature sets, each of which includes a central image feature and image features similar to the central image feature; The retrieval module comprises: A feature retrieval unit, configured to search the image feature library for a central image feature whose similarity with the third image feature satisfies a first similarity condition; A set determination unit, used to determine the feature set where the found central image feature belongs; The image retrieval unit is used to search for an image feature whose similarity with the third image feature satisfies a second similarity condition in the determined feature set, and determine the image corresponding to the searched image feature as a similar image to the target image.

21. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the training method of the feature extraction model as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the training method of the feature extraction model as described in any one of claims 1 to 10.

23. A computer program product comprising a computer program, characterized in that The computer program is loaded and executed by a processor to implement the operations performed by the method for training a feature extraction model as described in any one of claims 1 to 10.

Images