Image feature extraction method, training method, device, electronic device and medium

By training multiple types of images, the network generation and decoding process in the initial image feature extraction model is used to generate the target image feature extraction model, which solves the problem of insufficient model performance caused by limited data volume in the prior art, and achieves more efficient image feature extraction.

CN112580666BActive Publication Date: 2024-12-03BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011506995.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-18
Publication Date
2024-12-03
Estimated Expiration
2040-12-18

AI Technical Summary

Technical Problem

In the prior art, the image feature extraction model cannot learn more image features when the amount of data is limited, resulting in insufficient model performance.

Method used

By acquiring multiple types of image training data sets, the reference vector generation network, feature map generation network and decoding network are used in the initial image feature extraction model, and each type of image is decoded and corrected to generate the target image feature extraction model. .

Benefits of technology

This method not only saves training costs, but also increases the amount of data, improves the capabilities of the basic network, and enhances the accuracy and scope of application of image feature extraction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112580666B_ABST
    Figure CN112580666B_ABST
Patent Text Reader

Abstract

The present application discloses an image feature extraction method, training method, device, electronic device and medium, and relates to the field of artificial intelligence, specifically computer vision and deep learning technology. The training method is: according to the type of each image in the training data set, each image is input into the corresponding reference vector generation network in the initial image feature extraction model to obtain the reference vector of each type of image; the training data set is input into the feature map generation network to obtain the initial feature map of the training data set; using multiple decoding networks, each type of reference vector and initial feature map are decoded respectively to determine the predicted features of each type of image; according to the difference between the labeled features and the predicted features of each type of image, the reference vector generation network, the feature map generation network and the decoding network are corrected respectively to generate the target image feature extraction model. This method uses multiple types of data for simultaneous training, which saves costs and improves the capabilities of the basic network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, specifically computer vision and deep learning technology, and in particular to an image feature extraction method, training method, device, electronic device and medium. Background Art

[0002] In tasks such as image recognition and retrieval, the model's image feature extraction capability is particularly important. The model's image feature extraction capability determines the performance of image recognition models, image retrieval models, etc., and is particularly critical for large-scale online tasks.

[0003] In the related art, a feature extraction loss function is mainly used to train a feature extractor on a specific image data set, and the trained feature extractor is used to extract image features. Summary of the invention

[0004] The present application provides an image feature extraction method, training method, device, electronic equipment and medium.

[0005] According to one aspect of the present application, a method for training an image feature extraction model is provided, comprising:

[0006] Acquire a training data set, wherein the training data set includes multiple types of images and annotation features corresponding to each type of image;

[0007] According to the type of each image, the images in the training data set are respectively input into the corresponding reference vector generation network in the initial image feature extraction model to obtain the reference vector corresponding to each type of image;

[0008] Inputting the training data set into a feature map generation network in the initial image feature extraction model to obtain an initial feature map corresponding to the training data set;

[0009] Using a plurality of decoding networks in the initial image feature extraction model, respectively decoding the reference vector and the initial feature map corresponding to each type, so as to determine the prediction features corresponding to each type of image;

[0010] According to the difference between the labeled features and the predicted features corresponding to each type of image, the reference vector generation network, the feature map generation network and the decoding network are modified respectively to generate a target image feature extraction model.

[0011] According to another aspect of the present application, a method for extracting image features is provided, comprising:

[0012] Determine the target type of the image to be processed;

[0013] According to the target type, a corresponding target model is obtained, wherein the target model is generated by using the model training method described in the above-mentioned embodiment;

[0014] The image to be processed is input into the target model to obtain features of the image to be processed.

[0015] According to another aspect of the present application, a training device for an image feature extraction model is provided, comprising:

[0016] A first acquisition module is used to acquire a training data set, wherein the training data set includes multiple types of images and annotation features corresponding to each type of image;

[0017] A second acquisition module is used to input the images in the training data set into the corresponding reference vector generation network in the initial image feature extraction model according to the type of each image, so as to obtain the reference vector corresponding to each type of image;

[0018] A third acquisition module, used for inputting the training data set into the feature map generation network in the initial image feature extraction model to obtain an initial feature map corresponding to the training data set;

[0019] A determination module, used to respectively decode the reference vector and the initial feature map corresponding to each type by using a plurality of decoding networks in the initial image feature extraction model, so as to determine the prediction features corresponding to each type of image;

[0020] A generation module is used to modify the reference vector generation network, the feature map generation network and the decoding network according to the differences between the labeled features and the predicted features corresponding to each type of image, so as to generate a target image feature extraction model.

[0021] According to another aspect of the present application, there is provided a device for extracting image features, comprising:

[0022] A determination module, used for determining the target type to which the image to be processed belongs;

[0023] A first acquisition module is used to acquire a corresponding target model according to the target type, wherein the target model is generated by using the model training method described in the above-mentioned embodiment;

[0024] The second acquisition module is used to input the image to be processed into the target model to obtain the features of the image to be processed.

[0025] According to another aspect of the present application, there is provided an electronic device, including:

[0026] at least one processor; and

[0027] a memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the image feature extraction model or the image feature extraction method described in the above embodiments.

[0029] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, on which a computer program is stored, and the computer instructions are used to enable the computer to execute the training method of the image feature extraction model or the image feature extraction method described in the above embodiments.

[0030] In another embodiment of the present application, a computer program product is proposed, including a computer program, wherein when the computer program is executed by a processor, the training method of the image feature extraction model or the image feature extraction method described in the above embodiments is implemented.

[0031] Other effects of the above optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0033] Figure 1 A flowchart of a method for training an image feature extraction model provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of another method for training an image feature extraction model provided in an embodiment of the present application;

[0035] Figure 3 A flowchart of another method for training an image feature extraction model provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of an initial image feature extraction model provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a flow chart of a method for extracting image features provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of a flow chart of another method for extracting image features provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the structure of a training device for an image feature extraction model provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the structure of an image feature extraction device provided in an embodiment of the present application;

[0041] Fig. 9 It is a block diagram of an electronic device used to implement the training method of the image feature extraction model of the embodiment of the present application. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0043] The following describes the image feature extraction method, training method, device, electronic device and medium of the embodiments of the present application with reference to the accompanying drawings.

[0044] Artificial intelligence is a discipline that studies the use of computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0045] Computer vision is a science that studies how to make machines "see". It refers to using cameras and computer equipment to replace the human eye to identify, track and measure targets, and further perform graphic processing so that the computer equipment can process the images into images that are more suitable for human eye observation or transmission to instruments for detection.

[0046] Deep learning is a new research direction in the field of machine learning. Deep learning is the study of the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.

[0047] In the related technology, a feature extraction loss function is mainly used to train a feature extractor on a specific image data set, and the trained feature extractor is used to extract the features of the image. However, this extraction method is usually subject to data bottlenecks. Due to the limited amount of data, the model cannot learn more image features.

[0048] Based on this, an embodiment of the present application provides a method for training an image feature extraction model so as to perform training simultaneously by utilizing multiple types of images.

[0049] Figure 1 A flowchart of a method for training an image feature extraction model provided in an embodiment of the present application.

[0050] The training method of the image feature extraction model of the embodiment of the present application can be executed by the training device of the image feature extraction model provided by the embodiment of the present application. The device can be configured in an electronic device. By using multiple types of images to train the model at the same time, it not only saves training costs but also increases the amount of data, thereby increasing the capacity of the basic network.

[0051] like Figure 1 As shown, the training method of the image feature extraction model includes:

[0052] Step 101: Obtain a training data set, wherein the training data set includes multiple types of images and annotation features corresponding to each type of image.

[0053] In this embodiment, the training data set includes multiple types of images, for example, images of people, images of landscapes, images of buildings, etc., or other types, which are not limited in this embodiment. There are multiple images of each type.

[0054] In this embodiment, the annotation features may include, but are not limited to, image features such as color features, texture features, and shape features. The annotation features corresponding to each type of image may be obtained by synthesizing image features of the same type extracted from multiple images of each type. In other words, the annotation features corresponding to each type of image may be regarded as the same image features of each type of image.

[0055] Step 102: According to the type of each image, the images in the training data set are respectively input into the corresponding reference vector generation network in the initial image feature extraction model to obtain the reference vector corresponding to each type of image.

[0056] In this embodiment, the initial feature extraction model includes a feature map generation network, multiple reference vector generation networks and multiple decoding networks. Among them, each reference vector generation network can be the same or different, which can be determined according to the actual task. The number of reference vector generation networks and the number of decoding networks are equal to the number of image types in the training data set. Each image type can correspond to a reference vector generation network and a decoding network. The reference vector generation network can use a gated network.

[0057] For example, if the training data set includes 5 types of images, the initial feature extraction model may include 5 reference vector generation networks and 5 decoding networks.

[0058] In this embodiment, each image in the training data set can be input into a reference vector generation network corresponding to the type of each image according to the type to which each image belongs, so as to utilize the reference vector generation network corresponding to each image type to obtain the reference vector corresponding to each type of image.

[0059] For example, multiple images of each type A in the training data set are input into the reference vector generation network a, so as to use the reference vector generation network a to obtain the reference vector corresponding to the type A image.

[0060] Step 103: input the training data set into the feature map generation network in the initial image feature extraction model to obtain an initial feature map corresponding to the training data set.

[0061] In this embodiment, the feature map generation network can be regarded as the basic network of the initial feature extraction model. Each image in the training data set can be input into the feature map generation network, and the initial feature map of each image can be obtained by using the feature map generation network. Based on the initial feature map of each image, the initial feature map corresponding to the training data set can be obtained.

[0062] Wherein, based on the initial feature map of each image, the initial feature map corresponding to the training data set is obtained, and the initial feature map of each image in the training data set may be fused to obtain the initial feature map corresponding to the training data set.

[0063] In this embodiment, the feature map generation network is used to obtain an initial feature map corresponding to the training data set, and the initial feature map includes features of all types of images in the training data set.

[0064] Step 104, using multiple decoding networks in the initial image feature extraction model, respectively decode the reference vector and the initial feature map corresponding to each type to determine the prediction features corresponding to each type of image.

[0065] In this embodiment, the initial image feature extraction model includes multiple decoding networks, and each image type may correspond to one decoding network. After obtaining the reference vector corresponding to each type of image and the initial feature map corresponding to the training data set, the reference vector and the initial feature map of each type of image may be input into the corresponding decoding network, and decoded using the decoding network corresponding to each image type to obtain the prediction features corresponding to each type of image.

[0066] Step 105, according to the difference between the labeled features and the predicted features corresponding to each type of image, the reference vector generation network, the feature map generation network and the decoding network are modified respectively to generate a target image feature extraction model.

[0067] In this embodiment, after obtaining the prediction features corresponding to each type of image, the parameters of the feature map generation network, the reference vector generation network corresponding to each image type, and the decoding network can be adjusted according to the differences between the labeled features and the prediction features corresponding to each type of image, and the adjusted parameters can be used to continue training until the accuracy of the model meets the requirements, thereby generating a target image feature extraction model.

[0068] The target image feature extraction model includes image feature extraction models for multiple types of images, wherein the image feature extraction model corresponding to each type of image includes a feature map generation network, a reference vector generation network and a decoding network, and the image feature extraction model corresponding to each type of image shares the initial feature map generation network. It can be seen that the target image feature extraction model can extract features from multiple types of images.

[0069] Optionally, a deep learning method may be used to train the initial image feature extraction model to obtain a target image feature extraction model.

[0070] The training method of the image feature extraction model of the embodiment of the present application is to obtain a training data set, wherein the training data set includes multiple types of images and the annotation features corresponding to each type of image, and input the images in the training data set into the corresponding reference vector generation network in the initial image feature extraction model according to the type of each image to obtain the reference vector corresponding to each type of image, and input the training data set into the feature map generation network in the initial image feature extraction model to obtain the initial feature map corresponding to the training data set, and use the multiple decoding networks in the initial image feature extraction model to respectively decode the reference vector and the initial feature map corresponding to each type to determine the prediction features corresponding to each type of image, and according to the difference between the annotation features and the prediction features corresponding to each type of image, respectively correct the reference vector generation network, the feature map generation network and the decoding network to generate the target image feature extraction model. Therefore, by using multiple types of images for training at the same time, not only the training cost is saved, but also the amount of data is increased, so that the capacity of the basic network is increased.

[0071] In one embodiment of the present application, when determining the prediction features corresponding to each type of image, the initial feature map corresponding to the training data set can be decomposed to obtain the initial feature map corresponding to each type of image, and the initial feature map can be modified based on the reference vector corresponding to each type of image, and the modified initial feature map can be decoded to determine the prediction features corresponding to each type of image. Figure 2 To explain, Figure 2 A flowchart of another method for training an image feature extraction model provided in an embodiment of the present application.

[0072] like Figure 2 As shown, the above-mentioned multiple decoding networks in the initial image feature extraction model are used to decode the reference vector and the initial feature map corresponding to each type of image respectively to determine the prediction features corresponding to each type of image, including:

[0073] Step 201, decomposing the initial feature map to obtain an initial feature map corresponding to each type of image.

[0074] Since the initial feature map corresponding to the training data set includes feature maps of all types of images in the training data set, in this embodiment, the initial feature map corresponding to the training data set can be decomposed to obtain the initial feature map corresponding to each type of image.

[0075] Step 202: Based on the reference vector corresponding to each type of image, the initial feature map corresponding to each type of image is corrected to obtain a target feature map corresponding to each type of image.

[0076] In this embodiment, for each type of image, the reference vector corresponding to each type of image can be multiplied by the initial feature map corresponding to each type of image, so as to use the reference vector of each type of image to correct the initial feature map of each type of image, and the multiplication result can be used as the target feature map corresponding to each type of image.

[0077] Step 203, using multiple decoding networks in the initial image feature extraction model, decode the target feature map corresponding to each type of image to determine the prediction features corresponding to each type of image.

[0078] In this embodiment, after obtaining the target feature map corresponding to each type of image, the target feature map of each type of image can be input into the corresponding decoding network, and the target feature map can be decoded using the corresponding decoding network to obtain the prediction features corresponding to each type of image.

[0079] After obtaining the target feature map corresponding to each type of image, the decoding network corresponding to each type of image is used to decode the initial features corresponding to each type of image to obtain the predicted features. After obtaining the predicted features corresponding to each type of image, the feature map generation network, the corresponding reference vector generation network and the decoding network are corrected using the difference between the predicted features and the labeled features corresponding to each type of image to obtain the target image feature extraction model.

[0080] In an embodiment of the present application, the initial feature map corresponding to the training data set is decomposed to obtain the initial feature map corresponding to each type of image, the reference vector corresponding to each type of image is used to correct the initial feature map corresponding to each type, the target feature map corresponding to each type is obtained, and the target feature map corresponding to each type of image is decoded, the prediction features corresponding to each type of image are determined, training is performed based on the prediction features and annotation features corresponding to each type of image, and a target feature extraction model is generated, thereby achieving simultaneous training using multiple types of images, which not only saves training costs but also increases the amount of data, thereby increasing the capacity of the basic network.

[0081] In one embodiment of the present application, when the initial feature map corresponding to the training data set is decomposed, the initial feature map corresponding to each type of image can be decomposed according to the dimension of each type of image in the feature map generation network. Figure 3 To explain, Figure 3 A flowchart of another method for training an image feature extraction model provided in an embodiment of the present application.

[0082] like Figure 3 As shown, the above-mentioned multiple decoding networks in the initial image feature extraction model are used to decode the reference vector and the initial feature map corresponding to each type of image respectively to determine the prediction features corresponding to each type of image, including:

[0083] Step 301, according to the dimension corresponding to each type of image in the feature map generation network, determine the decomposition method corresponding to the initial feature map.

[0084] In this embodiment, the position of each type of image in the initial feature map corresponding to the training data set can be determined according to the dimension corresponding to each type of image in the feature map generation network, and the decomposition method corresponding to the initial feature map corresponding to the training data set can be determined according to the position corresponding to each type of image.

[0085] Optionally, the number or size of dimensions corresponding to each type of image in the feature map generation network may be the same or different.

[0086] Step 302: Decompose the initial feature map according to the decomposition method to obtain an initial feature map corresponding to each type of image.

[0087] In this embodiment, the initial feature map corresponding to the training data set can be decomposed into multiple feature maps according to the decomposition method corresponding to the initial feature map, such as the position corresponding to each type of image in the initial feature map, so as to obtain the initial feature map corresponding to each type of image.

[0088] Step 303: Based on the reference vector corresponding to each type of image, the initial feature map corresponding to each type of image is corrected to obtain a target feature map corresponding to each type of image.

[0089] Step 304, using multiple decoding networks in the initial image feature extraction model, decode the target feature map corresponding to each type of image to determine the prediction features corresponding to each type of image.

[0090] In this embodiment, step 303 - step 304 are similar to the above-mentioned step 202 - step 203 , and thus will not be described in detail here.

[0091] In an embodiment of the present application, when the initial feature map is decomposed to obtain the initial feature map corresponding to each type of image, the initial feature map corresponding to the training data is decomposed according to the dimension corresponding to each type of image in the feature map generation network, so that the initial feature map corresponding to each type of image can be obtained, so that the predicted features can be obtained by using the initial feature map corresponding to each type of image and the reference vector, and the predicted features and the labeled features are used to train the target image feature extraction model.

[0092] Combine the following Figure 4 The above-mentioned embodiment is described. Figure 4 A schematic diagram of an initial image feature extraction model provided in an embodiment of the present application. Figure 4 In this paper, we take the example that the training dataset includes two types of images.

[0093] like Figure 4 As shown, the initial image feature extraction model includes two reference vector generation networks, a feature map generation network and two decoding networks. Type A images and type B images are respectively input into the two reference vector generation networks to obtain the reference vectors corresponding to the two types of images, and type A images and type B images are input into the feature map generation network to obtain the initial feature map corresponding to the training data set, and the initial feature map of the training data set is decomposed to obtain the initial feature map corresponding to type A images and the initial feature map corresponding to type B images.

[0094] After obtaining the initial feature maps corresponding to type A image and type B image respectively, the initial feature map corresponding to type A image is multiplied by the reference vector to obtain the target feature map corresponding to type A image, the initial feature map corresponding to type B image is multiplied by the reference vector to obtain the target feature map corresponding to type B image, the target feature map corresponding to type A image is input into the corresponding decoding network to obtain the predicted features of type A image, and the target feature map corresponding to type B image is input into the corresponding decoding network to obtain the predicted features of type B image.

[0095] After obtaining the prediction features corresponding to the two types of images, the difference between the prediction features and the annotation features corresponding to the two types of images can be used to correct the reference vector generation network, the feature map generation network and the decoding network to generate a target image feature extraction model.

[0096] In order to improve the accuracy and training effect of the model, in one embodiment of the present application, before obtaining the training data set, the training data may be preprocessed to obtain the target training data set.

[0097] In this embodiment, each image in the training data set may be preprocessed to obtain a corresponding image, and a target training data set may be obtained based on the preprocessed image. The preprocessing includes but is not limited to partial magnification, image folding, etc., and may be any one or more of the image preprocessing methods.

[0098] In this embodiment, the target training data set includes multiple types of images, and the number of images of each type is the same.

[0099] In the embodiment of the present application, after obtaining the training data set, the training data set is preprocessed to obtain a target training data set, wherein the target training data set contains the same number of images of each type. Thus, by preprocessing, a target training data set containing the same number of images of each type is obtained, and the target training data set is used for model training, which not only increases the amount of training data, but also avoids the problem of low accuracy of the feature extraction model for one or several types of images.

[0100] Figure 5 A flowchart of an image feature extraction method provided in an embodiment of the present application. The image feature extraction method of the embodiment of the present application can be performed by an image feature extraction device of the embodiment of the present application, which can be configured in an electronic device to extract image features using a model corresponding to the type of image to be processed.

[0101] like Figure 5 As shown, the method for extracting the image features includes:

[0102] Step 501: determine the target type to which the image to be processed belongs.

[0103] In this embodiment, the image to be processed can be identified, and the target type of the image to be processed can be determined according to the identified objects. For example, if the identified objects include people, the type of the image to be processed can be considered to be a person type.

[0104] Alternatively, the target type of the image to be processed may be determined according to a preset image type.

[0105] Step 502: Obtain the corresponding target model according to the target type.

[0106] In this embodiment, the target image feature extraction model obtained by training using the training method of the image feature extraction model described in the above embodiment includes image feature extraction models for multiple types of images, and each type of image has a corresponding image feature extraction model. After determining the target type to which the image to be processed belongs, the image feature extraction model corresponding to the target type can be obtained according to the target type, which is referred to as the target model herein.

[0107] Step 503: input the image to be processed into the target model to obtain the features of the image to be processed.

[0108] After obtaining a target model corresponding to the type of the image to be processed, the image to be processed may be input into the target model, and the target model may be used to perform feature extraction on the image to be processed to obtain features of the image to be processed.

[0109] The image feature extraction method of the embodiment of the present application determines the target type to which the image to be processed belongs, and obtains the corresponding target model according to the target type, wherein the target model is generated by the image feature extraction model training method described in the above embodiment, and the image to be processed is input into the target model to obtain the features of the image to be processed. Therefore, by using the image feature extraction model generated by the model training method described in the above embodiment, the features of different types of images can be obtained, and the application range is wide.

[0110] Figure 6 A schematic flow chart of another method for extracting image features provided in an embodiment of the present application.

[0111] In this embodiment, in the image feature extraction model generated by the above model training method, the image feature extraction model corresponding to each type of image includes a reference vector generation network, an initial feature map generation network and a decoding network, wherein the image feature extraction model corresponding to each type of image shares the initial feature map generation network. Then, the target model also includes a reference vector generation network, an initial feature map generation network and a decoding network.

[0112] like Figure 6 As shown, the method for extracting the image features includes:

[0113] Step 601, determining the target type to which the image to be processed belongs.

[0114] Step 602: Obtain the corresponding target model according to the target type.

[0115] In this embodiment, step 601 - step 602 are similar to the above-mentioned step 501 - step 502, and thus will not be described in detail here.

[0116] Step 603: Input the image to be processed into the reference vector generation network and the initial feature map generation network respectively, so as to obtain the reference vector and the initial feature map corresponding to the image to be processed respectively.

[0117] In this embodiment, the image to be processed can be input into the reference vector generation network and the initial feature map generation network respectively. The reference vector generation network can be used to obtain the reference vector corresponding to the image to be processed, and the initial feature map generation network can be used to obtain the initial feature map corresponding to the image to be processed.

[0118] Step 604: based on the reference vector corresponding to the image to be processed, the initial feature map is corrected to obtain a target feature map corresponding to the image to be processed.

[0119] In this embodiment, the initial feature map can be corrected by multiplying the reference vector corresponding to the image to be processed with the initial feature map, and the multiplication result can be used as the target feature map corresponding to the image to be processed.

[0120] In this embodiment, the initial feature map is corrected using the reference vector corresponding to the image to be processed, which can improve the accuracy of the extracted image features.

[0121] Step 605: Decode the target feature map using a decoding network to obtain features of the image to be processed.

[0122] After obtaining the target feature map of the image to be processed, the target feature map can be input into the decoding network of the target model, and the target feature map can be decoded by using the decoding network, and the decoding network outputs the features of the image to be processed.

[0123] In an embodiment of the present application, the target model includes a reference vector generation network, an initial feature map generation network and a decoding network. By utilizing the reference vector generation network and the initial feature map generation network, the reference vector and the initial feature map of the image to be processed are respectively obtained, the initial feature map is corrected using the reference vector, and the corrected target feature map is decoded using the decoding network to obtain the features of the image to be processed, thereby improving the accuracy of the extracted image features.

[0124] In order to implement the above embodiment, the embodiment of the present application also proposes a training device for an image feature extraction model. Figure 7 A schematic diagram of the structure of a training device for an image feature extraction model provided in an embodiment of the present application.

[0125] like Figure 7As shown, the training device 700 of the image feature extraction model includes: a first acquisition module 710, a second acquisition module 720, a third acquisition module 730, a determination module 740, and a generation module 750.

[0126] A first acquisition module 710 is used to acquire a training data set, wherein the training data set includes multiple types of images and annotation features corresponding to each type of image;

[0127] A second acquisition module 720 is used to input the images in the training data set into the corresponding reference vector generation network in the initial image feature extraction model according to the type of each image, so as to obtain the reference vector corresponding to each type of image;

[0128] A third acquisition module 730 is used to input the training data set into the feature map generation network in the initial image feature extraction model to obtain an initial feature map corresponding to the training data set;

[0129] A determination module 740 is used to use multiple decoding networks in the initial image feature extraction model to decode the reference vector and the initial feature map corresponding to each type, so as to determine the prediction features corresponding to each type of image;

[0130] The generation module 750 is used to modify the reference vector generation network, the feature map generation network and the decoding network according to the difference between the annotation features and the predicted features corresponding to each type of image, so as to generate a target image feature extraction model.

[0131] In a possible implementation of the embodiment of the present application, the determination module 740 includes:

[0132] A first acquisition unit is used to decompose the initial feature map to obtain an initial feature map corresponding to each type of image;

[0133] A second acquisition unit is used to correct the initial feature map corresponding to each type of image based on the reference vector corresponding to each type of image, so as to obtain a target feature map corresponding to each type of image;

[0134] The determination unit is used to decode the target feature map corresponding to each type of image by using multiple decoding networks in the initial image feature extraction model to determine the prediction features corresponding to each type of image.

[0135] In a possible implementation of the embodiment of the present application, the first acquisition unit is used to:

[0136] Determine the decomposition method corresponding to the initial feature map according to the corresponding dimension of each type of image in the feature map generation network;

[0137] According to the decomposition method, the initial feature map is decomposed to obtain the initial feature map corresponding to each type of image.

[0138] In a possible implementation manner of the embodiment of the present application, the preprocessing module is used to preprocess the training data set to obtain a target training data set, wherein the number of images of various types contained in the target training data set is the same.

[0139] It should be noted that the explanation of the aforementioned embodiment of the training method of the image feature extraction model is also applicable to the training device of the image feature extraction model of this embodiment, so it will not be repeated here.

[0140] The training device of the image feature extraction model of the embodiment of the present application obtains a training data set, wherein the training data set includes multiple types of images and the annotation features corresponding to each type of image, and inputs the images in the training data set into the corresponding reference vector generation network in the initial image feature extraction model according to the type of each image to obtain the reference vector corresponding to each type of image, and inputs the training data set into the feature map generation network in the initial image feature extraction model to obtain the initial feature map corresponding to the training data set, and uses multiple decoding networks in the initial image feature extraction model to decode the reference vector and initial feature map corresponding to each type respectively to determine the prediction features corresponding to each type of image, and according to the difference between the annotation features and the prediction features corresponding to each type of image, the reference vector generation network, the feature map generation network and the decoding network are respectively corrected to generate the target image feature extraction model. Therefore, by using multiple types of images for training at the same time, not only the training cost is saved, but also the amount of data is increased, so that the capacity of the basic network is increased.

[0141] In order to implement the above embodiment, the embodiment of the present application also proposes a device for extracting image features. Figure 8 A schematic diagram of the structure of an image feature extraction device provided in an embodiment of the present application.

[0142] like Figure 8 As shown, the image feature extraction device includes: a determination module 810, a first acquisition module 820, and a second acquisition module 830.

[0143] A determination module 810 is used to determine the target type to which the image to be processed belongs;

[0144] A first acquisition module 820 is used to acquire a corresponding target model according to the target type, wherein the target model is generated by using the above-mentioned image feature extraction model training method;

[0145] The second acquisition module 830 is used to input the image to be processed into the target model to obtain the features of the image to be processed.

[0146] In a possible implementation of the embodiment of the present application, the second acquisition module 830 is used to:

[0147] Inputting the image to be processed into the reference vector generation network and the initial feature map generation network respectively, so as to obtain the reference vector and the initial feature map corresponding to the image to be processed respectively;

[0148] Based on the reference vector corresponding to the image to be processed, the initial feature map is corrected to obtain the target feature map corresponding to the image to be processed;

[0149] The target feature map is decoded using the decoding network to obtain the features of the image to be processed.

[0150] It should be noted that the explanation of the above-mentioned embodiment of the method for extracting image features is also applicable to the image feature extraction device of this embodiment, so it will not be repeated here.

[0151] The image feature extraction device of the embodiment of the present application determines the target type to which the image to be processed belongs, and obtains the corresponding target model according to the target type, wherein the target model is generated by the image feature extraction model training method described in the above embodiment, and the image to be processed is input into the target model to obtain the features of the image to be processed. Therefore, by using the image feature extraction model generated by the model training method described in the above embodiment, the features of different types of images can be obtained, and the application range is wide.

[0152] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0153] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0154] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 902 or a computer program loaded from a storage unit 908 to a RAM (Random Access Memory) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An I / O (Input / Output) interface 905 is also connected to the bus 904.

[0155] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0156] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the training method of the image feature extraction model. For example, in some embodiments, the training method of the image feature extraction model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method of the image feature extraction model described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the training method of the image feature extraction model in any other appropriate manner (eg, by means of firmware).

[0157] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0159] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0161] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0162] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0163] It should be noted that the electronic device of the image feature extraction method is similar to the electronic device of the above-mentioned image feature extraction model training method, so it will not be repeated here.

[0164] In order to implement the above embodiments, the embodiments of the present application also propose a computer program product, including a computer program, wherein when the computer program is executed by a processor, it implements the training method of the image feature extraction model or the image feature extraction method described in the above embodiments.

[0165] According to the technical solution of the embodiment of the present application, it relates to the field of artificial intelligence, specifically computer vision and deep learning technology, by obtaining a training data set, according to the type of each image, the images in the training data set are respectively input into the corresponding reference vector generation network in the initial image feature extraction model to obtain the reference vector corresponding to each type of image, and the training data set is input into the feature map generation network in the initial image feature extraction model to obtain the initial feature map corresponding to the training data set, and the reference vector and initial feature map corresponding to each type are respectively decoded by using multiple decoding networks in the initial image feature extraction model to determine the prediction features corresponding to each type of image, and the reference vector generation network, feature map generation network and decoding network are respectively corrected according to the difference between the annotation features and the prediction features corresponding to each type of image to generate the target image feature extraction model. Therefore, by using multiple types of images for training at the same time, not only the training cost is saved, but also the amount of data is increased, so that the capacity of the basic network is increased.

[0166] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0167] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A training method for an image feature extraction model, comprising: Acquire a training data set, wherein the training data set includes multiple types of images and annotation features corresponding to each type of image; According to the type of each image, the images in the training data set are respectively input into the corresponding reference vector generation network in the initial image feature extraction model to obtain the reference vector corresponding to each type of image; Inputting the training data set into a feature map generation network in the initial image feature extraction model to obtain an initial feature map corresponding to the training data set; Using a plurality of decoding networks in the initial image feature extraction model, respectively decoding the reference vector and the initial feature map corresponding to each type, so as to determine the prediction features corresponding to each type of image; According to the difference between the labeled features and the predicted features corresponding to each type of image, the reference vector generation network, the feature map generation network and the decoding network are modified respectively to generate a target image feature extraction model.

2. The method of claim 1, wherein: The method of using a plurality of decoding networks in the initial image feature extraction model to respectively decode the reference vector and the initial feature map corresponding to each type of image to determine the prediction features corresponding to each type of image includes: Decomposing the initial feature map to obtain an initial feature map corresponding to each type of image; Based on the reference vector corresponding to each type of image, the initial feature map corresponding to each type of image is corrected to obtain the target feature map corresponding to each type of image; The target feature maps corresponding to each type of image are decoded by using multiple decoding networks in the initial image feature extraction model to determine the prediction features corresponding to each type of image.

3. The method of claim 2, wherein: Decomposing the initial feature map to obtain the initial feature map corresponding to each type of image includes: Determining a decomposition method corresponding to the initial feature map according to the dimensions corresponding to each type of image in the feature map generation network; According to the decomposition method, the initial feature map is decomposed to obtain the initial feature map corresponding to each type of image.

4. The method according to any one of claims 1 to 3, wherein: After obtaining the training data set, the method further includes: The training data set is preprocessed to obtain a target training data set, wherein the target training data set contains the same number of images of various types.

5. A method for extracting image features, comprising: Determine the target type of the image to be processed; According to the target type, obtaining a corresponding target model, wherein the target model is generated by using the model training method according to any one of claims 1 to 4; The image to be processed is input into the target model to obtain features of the image to be processed.

6. The method of claim 5, wherein: The target model includes a reference vector generation network, an initial feature map generation network and a decoding network, and the step of inputting the image to be processed into the target model to obtain the features of the image to be processed includes: Inputting the image to be processed into the reference vector generation network and the initial feature map generation network respectively, so as to obtain the reference vector and the initial feature map corresponding to the image to be processed respectively; Based on the reference vector corresponding to the image to be processed, the initial feature map is corrected to obtain a target feature map corresponding to the image to be processed; The target feature map is decoded using the decoding network to obtain the features of the image to be processed.

7. A training device for an image feature extraction model, comprising: A first acquisition module is used to acquire a training data set, wherein the training data set includes multiple types of images and annotation features corresponding to each type of image; A second acquisition module is used to input the images in the training data set into the corresponding reference vector generation network in the initial image feature extraction model according to the type of each image, so as to obtain the reference vector corresponding to each type of image; A third acquisition module, used for inputting the training data set into the feature map generation network in the initial image feature extraction model to obtain an initial feature map corresponding to the training data set; A determination module, used to respectively decode the reference vector and the initial feature map corresponding to each type by using a plurality of decoding networks in the initial image feature extraction model, so as to determine the prediction features corresponding to each type of image; A generation module is used to modify the reference vector generation network, the feature map generation network and the decoding network according to the differences between the annotation features and the predicted features corresponding to each type of image, so as to generate a target image feature extraction model.

8. The device according to claim 7, wherein: The determining module comprises: A first acquisition unit, configured to decompose the initial feature map to obtain an initial feature map corresponding to each type of image; A second acquisition unit is used to correct the initial feature map corresponding to each type of image based on the reference vector corresponding to each type of image, so as to obtain a target feature map corresponding to each type of image; A determination unit is used to decode the target feature map corresponding to each type of image by using multiple decoding networks in the initial image feature extraction model to determine the prediction features corresponding to each type of image.

9. The device of claim 8, wherein: The first acquiring unit is used to: Determining a decomposition method corresponding to the initial feature map according to the dimensions corresponding to each type of image in the feature map generation network; According to the decomposition method, the initial feature map is decomposed to obtain the initial feature map corresponding to each type of image.

10. The device according to any one of claims 7 to 9, further comprising: The preprocessing module is used to preprocess the training data set to obtain a target training data set, wherein the target training data set contains the same number of images of various types.

11. A device for extracting image features, comprising: A determination module, used for determining the target type to which the image to be processed belongs; A first acquisition module, configured to acquire a corresponding target model according to the target type, wherein the target model is generated by using the model training method according to any one of claims 1 to 4; The second acquisition module is used to input the image to be processed into the target model to obtain the features of the image to be processed.

12. The device of claim 11, wherein: The second acquisition module is used to: Inputting the image to be processed into the reference vector generation network and the initial feature map generation network respectively, so as to obtain the reference vector and the initial feature map corresponding to the image to be processed respectively; Based on the reference vector corresponding to the image to be processed, the initial feature map is corrected to obtain a target feature map corresponding to the image to be processed; The target feature map is decoded using the decoding network to obtain the features of the image to be processed.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the image feature extraction model described in any one of claims 1-4, or the image feature extraction method described in any one of claims 5-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the training method of the image feature extraction model described in any one of claims 1-4, or the image feature extraction method described in any one of claims 5-6.

15. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, it implements the training method of the image feature extraction model described in any one of claims 1-4, or the image feature extraction method described in any one of claims 5-6.