Image processing methods, apparatus, computing devices and media

Abstract

This application provides an image processing method, apparatus, computing device, and medium to improve the recognition accuracy of the network while ensuring low computational and storage costs through transfer learning, thereby improving the classification accuracy of images. The image processing method includes: acquiring first features at multiple levels of the image to be processed through a backbone network; for any level, performing attention mechanism processing on multiple domain center vectors and the first features of the current level through an attention mechanism network to obtain second features; obtaining target features based on the first and second features of the last level; and obtaining the classification result of the image to be processed based on the target features through a classification network.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to an image processing method, apparatus, computing device, and medium. Background Technology

[0002] Transfer learning is a machine learning method that transfers knowledge learned from a source domain to a target domain, aiming to use existing knowledge to learn new knowledge. Generally, transfer learning typically involves first training a model based on data from the source domain to obtain a trained general network, and then fine-tuning the general network based on data from the target domain to obtain a network that meets the requirements.

[0003] In practical applications, the computational and storage costs of a network and its recognition accuracy are important metrics that need to be considered. Therefore, there is an urgent need for an image processing method that can improve the recognition accuracy of a network while maintaining low computational and storage costs through transfer learning, thereby improving the classification accuracy of images. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this application provides an image processing method, apparatus, computing device, and medium.

[0005] According to a first aspect of the embodiments of this application, an image processing method is provided, the method comprising:

[0006] The backbone network obtains the first features of multiple levels of the image to be processed. The first features of the first level are obtained by performing convolution operation on the image features of the image to be processed. The first features of non-first levels are obtained by performing convolution operation on the first features of the previous level. The backbone network is trained based on the first training set.

[0007] For any level, the attention mechanism network processes the multiple domain center vectors and the first feature of the level to obtain the second feature. The attention mechanism network is trained on the first training set. The domain center vectors of each level are obtained by clustering the sample features of the corresponding level. The sample features are obtained by the backbone network extracting features from the samples in the first training set.

[0008] The target features are obtained based on the first and second features of the last layer, and the classification network is used to obtain the classification result of the image to be processed based on the target features. The classification network is trained on the second training set.

[0009] According to a second aspect of the embodiments of this application, an image processing apparatus is provided, the apparatus comprising:

[0010] The feature acquisition module is used to acquire first features of multiple levels of the image to be processed through the backbone network. The first features of the first level in the multiple levels are obtained by performing convolution operation on the image features of the image to be processed, and the first features of non-first levels are obtained by performing convolution operation on the first features of the previous level. The backbone network is trained based on the first training set.

[0011] The feature acquisition module is also used to process the multiple domain center vectors and the first feature of any level through the attention mechanism network to obtain the second feature. The attention mechanism network is trained based on the first training set. The domain center vectors of each level are obtained by clustering the sample features of the corresponding level. The sample features are obtained by the backbone network extracting features from the samples in the first training set.

[0012] The feature acquisition module is also used to acquire target features based on the first and second features of the last level;

[0013] The result acquisition module is used to obtain the classification result of the image to be processed based on the target features through the classification network, wherein the classification network is trained based on the second training set.

[0014] According to a third aspect of the embodiments of this application, a computing device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the operations performed by the image processing method described above.

[0015] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a program is stored, and the program is executed by a processor to perform the operations performed by the above-described image processing method.

[0016] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the operations performed by the image processing method described above.

[0017] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0018] Having obtained the first features of multiple levels of the image to be processed through the backbone network, for each level, an attention mechanism network is used to process the multiple domain center vectors and the first features of the current level to obtain the second features. This allows the target features to be obtained based on the first and second features of the last level. A classification network then uses these target features to obtain the classification result of the image to be processed, thus achieving image processing. The backbone network and the attention mechanism network are trained on the first training set, while the classification network is trained on the second training set. This minimizes the need for excessive training data, ensuring the implementation of transfer learning. This allows for more efficient acquisition of a network specific to the scene corresponding to the second training set, building upon the existing training. Furthermore, the introduction of feature processing based on the attention mechanism and domain center vectors allows the network to focus more on features beneficial for classification, thereby improving image classification accuracy.

[0019] Furthermore, this application employs multiple clustering methods, including semantic and feature-based clustering, to divide the domain, enabling downstream tasks to draw upon upstream knowledge as much as possible. Additionally, this application constructs an auxiliary network at the same scale as each backbone network. This auxiliary network does not affect or modify the backbone network when extracting image features, thus improving the accuracy of feature extraction results. Moreover, this application obtains a fusion vector based on the attention score (i.e., the query vector) of the domain center vector and feature map. This fusion vector contains rich knowledge learned from upstream tasks, allowing the image classification process to focus more intently on regions truly beneficial for classification, thereby improving the accuracy of the classification results.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0022] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment of this application.

[0023] Figure 2 This is a schematic diagram of the network structure of an image processing network according to an exemplary embodiment of this application.

[0024] Figure 3 This is a block diagram of an image processing apparatus according to an exemplary embodiment of this application.

[0025] Figure 4 This is a schematic diagram of the structure of a computing device according to an exemplary embodiment of this application. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed herein.

[0027] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “described,” and “the” as used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0029] This application provides an image processing method for classifying an image to be processed based on a network trained by transfer learning, so as to obtain a classification label for the image to be processed, which can be used to indicate the category of objects included in the image to be processed.

[0030] The image to be processed can be of any type, such as an animal image, a person image, or a traffic scene monitoring image. This application does not limit the specific type of image to be processed. That is, the image processing method provided in this application can be used to classify any type of image to obtain classification labels for any type of image.

[0031] Optionally, after obtaining the classification label of the image to be processed, other computer vision tasks can be performed based on the obtained classification label. For example, high-level computer vision tasks such as object detection, image segmentation, object tracking, behavior analysis, and face recognition can be performed based on the obtained classification label. This application does not limit the specific type of computer vision task to be performed subsequently.

[0032] The above is merely an illustrative description of the application scenarios of this application and does not constitute a limitation on the application scenarios of this application. In many other possible implementations, this application can be applied to various other processes involving other data classifications. For example, the ideas provided in this application can also be used to process the text to be processed.

[0033] The above image processing method can be executed by a computing device, which can be a server, such as a single server, multiple servers, a server cluster, a cloud computing platform, etc. Optionally, the computing device can also be a terminal device, such as a mobile phone, tablet computer, game console, portable computer, desktop computer, advertising machine, all-in-one machine, etc. This application does not limit the specific type of computing device.

[0034] The above is a description of the application scenarios of this application. Next, the image processing method provided by this application will be described in detail with reference to the embodiments of this specification.

[0035] like Figure 1 As shown, Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment of this application, the method comprising the following steps:

[0036] Step 101: Obtain the first features of multiple levels of the image to be processed through the backbone network. The first features of the first level in the multiple levels are obtained by performing convolution operation on the image features of the image to be processed. The first features of non-first levels are obtained by performing convolution operation on the first features of the previous level. The backbone network is trained based on the first training set.

[0037] The first training set can be a general publicly available machine learning dataset. That is, the first training set can encompass sample images from multiple business domains and of various specifications, as long as the classification labels of the images included in the first training set are known. For example, the first training set can include multiple sample images of different types, sizes, and resolutions. If the classification labels of these multiple sample images are known, then all of these sample images in the first training set can be used as training images for training the backbone network, allowing the backbone network to be trained based on these multiple sample images. It should be noted that the term "non-first layer" refers to any or a specified layer other than the first layer among multiple layers.

[0038] Step 102: For any level, the attention mechanism network is used to process the multiple domain center vectors and the first feature of the current level to obtain the second feature. The attention mechanism network is trained on the first training set. The domain center vectors of each level are obtained by clustering the sample features of the corresponding level. The sample features are obtained by the backbone network extracting features from the samples in the first training set.

[0039] Each level can correspond to multiple domain center vectors. For any level, these multiple domain center vectors can be used to indicate the clustering centers of the sample features of the corresponding level.

[0040] Step 103: Obtain target features based on the first and second features of the last level, and obtain the classification result of the image to be processed based on the target features through a classification network. The classification network is trained based on the second training set.

[0041] It should be noted that the second training set can be a private dataset, or a pre-constructed image dataset tailored to the target data processing task. The sample images included in the second training set should correspond to the same business domain. For example, if the target data processing task is animal recognition, the images included in the second training set must be animal images, and the classification labels (i.e., animal categories) corresponding to the images in the second training set must be known. Additionally, it should be noted that the second training set contains a smaller amount of data compared to the first training set.

[0042] This application introduces a feature processing procedure based on attention mechanisms and domain center vectors during network processing. After obtaining first features at multiple levels of the image to be processed through the backbone network, for each level, an attention mechanism network processes the multiple domain center vectors and first features of the current level to obtain second features. This allows the acquisition of target features based on the first and second features of the last level, and the classification network then obtains the classification result of the image to be processed based on the target features, thus achieving image processing. The backbone network and attention mechanism network are trained on a first training set, while the classification network is trained on a second training set. This minimizes the need for excessive training data, ensuring transfer learning and enabling more efficient acquisition of a network for the corresponding scene in the second training set based on the existing training. Furthermore, the introduction of feature processing based on attention mechanisms and domain center vectors allows the network to focus more on features beneficial for classification, thereby improving image classification accuracy.

[0043] After introducing the basic implementation process of this application, the various non-limiting implementation methods of this application will be described in detail below.

[0044] The backbone network and attention mechanism network are trained on the first training set, while the classification network is trained on both the first and second training sets to enable network training through transfer learning. It should be noted that the backbone network, attention mechanism network, and classification network can form an image processing network, allowing for the implementation of corresponding image processing procedures.

[0045] The training process of the network will be introduced first below.

[0046] In some embodiments, an initial image processing network, including an initial backbone network, an initial attention mechanism network, and an initial classification network, can be trained based on sample images in a first training set to obtain a trained backbone network, an attention mechanism network, and a classification network to be fine-tuned; then, the classification network to be fine-tuned is trained based on sample images in a second training set to obtain a trained classification network, thereby obtaining an image processing network including a trained backbone network, an attention mechanism network, and a classification network.

[0047] It should be noted that the sample data in the first training set can be pre-labeled with classification labels so that the network can be trained based on the labeled classification labels.

[0048] In one possible implementation, the first sample image from the first training set can be input into an initial image processing network. The initial backbone network, initial attention mechanism network, and initial classification network of the initial image processing network process the first sample image to obtain a predicted label for the first sample image. Then, based on a loss function indicating the difference between the predicted label and the classification label of the first sample image, the network parameters of the initial backbone network, initial attention mechanism network, and initial classification network are adjusted to obtain an image processing network with first-adjusted parameters. This first-adjusted image processing network includes the backbone network, attention mechanism network, and classification network. Then, the second sample image from the first training set is input into the first-adjusted image processing network. The backbone network, attention mechanism network, and classification network of the first-adjusted image processing network process the second sample image to obtain a predicted label for the second sample image. Then, based on a loss function indicating the difference between the predicted label and the classification label of the second sample image, the parameters of the first-adjusted backbone network, attention mechanism network, and classification network are adjusted. The network parameters are further adjusted to obtain a second-adjusted image processing network, which includes a backbone network, an attention mechanism network, and a classification network. Then, a third sample image from the first training set is input into this second-adjusted network. The backbone network, attention mechanism network, and classification network of this second-adjusted network process the third sample image to obtain its predicted label. Based on a loss function indicating the difference between the predicted and classification labels of the third sample image, the backbone network, attention mechanism network, and classification network are further adjusted to obtain a third-adjusted image processing network, which includes the third-adjusted backbone network, attention mechanism network, and classification network. This process continues until the training cutoff condition is met. The backbone network, attention mechanism network, and classification network of the image processing network that meet the training cutoff condition are then used as the trained backbone network, attention mechanism network, and classification network for parameter fine-tuning.

[0049] The loss function can be of any type, such as absolute loss function, log loss function, exponential loss function, etc. This application does not limit the specific type of loss function.

[0050] In addition, the training cutoff condition can be that the loss function meets a set condition, or the number of iterations reaches a set number, or all sample images in the first training set have been processed, etc. This application does not limit the specific content of the training cutoff condition.

[0051] Through the above process, the backbone network, attention mechanism network, and classification network can be trained in the source domain, resulting in a trained backbone network, attention mechanism network, and a classification network for parameter fine-tuning. The trained backbone network, attention mechanism network, and classification network for parameter fine-tuning already possess basic image classification capabilities. However, the classification accuracy for images in specific domains is not yet good enough. Therefore, after completing network training based on source domain data, the classification network can be further trained based on target domain data to improve the classification accuracy of the classification network for images in specific domains.

[0052] After training the network based on the source domain data, the network parameters of the classification network can be fine-tuned based on the target domain data.

[0053] It should be noted that since the first training set is often a general dataset in the field of machine learning, the image processing network trained on the first training set may not be able to meet the personalized needs of customers well. Therefore, a second training set can be set according to the actual technical requirements by means of transfer learning. For example, a special second training set can be built according to the actual image domain to be recognized, so that the image processing network can be trained on the sample data in the second training set to obtain an image processing network that can better meet the actual technical requirements.

[0054] Furthermore, it is important to emphasize that the attention mechanism network incorporates the domain center vector during feature processing, allowing the network to focus more on certain important regions to obtain features that better meet classification requirements. Therefore, when training the image processing network based on sample images in the second dataset, only the classification network of the image processing network needs to be trained. This means that the amount of training data in the second training set does not need to be too large to meet the requirements, thus ensuring that good prediction results can still be achieved using a small amount of data for training.

[0055] The sample data in the second training set can be pre-labeled with classification labels so that the network parameters of the classification network can be fine-tuned based on the labeled classification labels.

[0056] In one possible implementation, adjusting the parameters of the classification network based on sample data from the second training set can be achieved through the following process:

[0057] The first sample image from the second training set is input into the image processing network trained on the first training set. The backbone network, attention mechanism network, and classification network process the first sample image to obtain a predicted label. The network parameters of the classification network are then adjusted based on a loss function indicating the difference between the predicted and classification labels of the first sample image, resulting in a classification network with the first parameter adjustment. Next, the second sample image from the second training set is input into the image processing network. The backbone network, attention mechanism network, and classification network (the classification network in the image processing network at this point is the one with the first parameter adjustment) process the second sample image to obtain a predicted label. The network parameters of the classification network are then adjusted based on a loss function indicating the difference between the predicted and classification labels of the second sample image. The loss function is used to further adjust the network parameters of the classification network after the first parameter adjustment to obtain a classification network after the second parameter adjustment. Then, the third sample image from the second training set is input into the image processing network. The backbone network, attention mechanism network, and classification network (at this time, the classification network in the image processing network is the classification network after the second parameter adjustment) process the third sample image to obtain the predicted label of the third sample image. Based on the loss function indicating the difference between the predicted label and the classification label of the third sample image, the network parameters of the classification network after the second parameter adjustment are further adjusted to obtain a classification network after the third parameter adjustment. This process is repeated until the training cutoff condition is met. The classification network that meets the training cutoff condition is then considered as the classification network trained through the parameter fine-tuning process.

[0058] The loss function can be of any type, such as absolute loss function, log loss function, exponential loss function, etc. This application does not limit the specific type of loss function.

[0059] In addition, the training cutoff condition can be that the loss function meets a set condition, the number of iterations reaches a set number, or all sample images in the second training set have been processed, etc. This application does not limit the specific content of the training cutoff condition.

[0060] Through the above process, the classification network can be trained in the target domain, thereby obtaining an image processing network that better meets user needs.

[0061] The above embodiments, by employing transfer learning, can train the network's feature extraction capabilities using a general dataset and train its classification capabilities using a specific dataset. This achieves the effect of training an image processing network with limited data, enabling more efficient acquisition of an image processing network for the corresponding scene of the second training set based on the original training. The image processing network can then demonstrate better classification capabilities in the corresponding scene of the second training set, thereby improving the accuracy of image classification.

[0062] After training the image processing network through the above process, the trained image processing network can be deployed to the application so that the image to be processed can be processed using the image processing method provided in this application.

[0063] In one possible implementation, the backbone network, multiple levels of attention mechanism networks, and classification network can be encapsulated as an image processing network; wherein the backbone network may include multiple convolutional layers, and the attention mechanism networks may include multiple first attention mechanism networks and multiple second attention mechanism networks, with one convolutional layer of the backbone network corresponding to one first attention mechanism network and one second attention mechanism network.

[0064] Once an image processing network is available, it can be used to classify the images to be processed.

[0065] Optionally, in response to acquiring the image to be processed, the image to be processed is input into an image processing network, and the image to be processed is processed by the image processing network to obtain the classification result of the image to be processed. Other computer vision tasks, such as object detection, image segmentation, object tracking, behavior analysis, and face recognition, are then performed based on the classification result of the image to be processed.

[0066] The classification results obtained through image processing networks are highly accurate, which can improve the accuracy of subsequent computer vision tasks.

[0067] The process of classifying images to be processed using the image processing network corresponding to the image processing method provided in this application will be described in detail below.

[0068] In some embodiments, the backbone network may include multiple layers of convolutional layers (it should be noted that a layer can also be called a stage). The multiple layers of convolutional layers can be connected in series. For step 101, when obtaining the first features of multiple layers of the image to be processed through the backbone network, it can be achieved in the following way:

[0069] For the first-level convolutional layer, the image features are convolved to obtain the first-level convolutional features, which serve as the first features of the first level. For any convolutional layer other than the first level, the first features output by the previous level are convolved to obtain the current level convolutional features, which serve as the first features of the current level, thus obtaining multiple first features of non-first levels.

[0070] Taking a backbone network comprising N convolutional layers as an example, the image features of the image to be processed are first convolved through the first convolutional layer to obtain the first feature of the first layer; then, the first feature of the first layer is further convolved through the second convolutional layer to obtain the first feature of the second layer; then, the first feature of the second layer is further convolved through the third convolutional layer to obtain the first feature of the third layer, and so on, until finally, the first feature of the (N-1)th layer is convolved through the Nth convolutional layer to obtain the first feature of the Nth layer. Here, N can be an integer greater than or equal to 2, and this application does not limit the specific value of N.

[0071] The above process yields the first features corresponding to different dimensions, allowing for processing based on these first features to achieve the effect of comprehensively determining the final features by integrating multiple dimensions, thereby improving the accuracy of the feature processing process.

[0072] After obtaining the first features at multiple levels through the above process, subsequent processing can be carried out based on the obtained first features at multiple levels.

[0073] In some embodiments, the attention mechanism network may include a first attention mechanism network (or Attention network) and a second attention mechanism network (or Domain Attention network). For step 102, for any level, when the second feature is obtained by performing attention mechanism processing on multiple domain center vectors and the first feature of the current level through the attention mechanism network, it can be achieved through the following steps:

[0074] Step 1: Based on the multiple domain center vectors of the current level and the first feature of the current level, obtain the query vector through the first attention mechanism network of the current level.

[0075] In one possible implementation, the query vector (Q) can be obtained through the following process:

[0076] Step 1: Perform global pooling on the first feature of the current level to obtain the feature vector corresponding to the first feature of the current level.

[0077] It should be noted that by performing global pooling on the first feature, the multidimensional first feature can be converted into a feature vector, thereby making the dimension of the feature vector and the domain center vector consistent, so as to ensure that the feature vector and the domain center vector can be processed through the first attention mechanism network.

[0078] Step 2: Input the feature vector corresponding to the first feature of the current level and the multiple domain center vectors of the current level into the first attention mechanism network of the current level. Through the first attention mechanism network of the current level, output the query vector of the current level.

[0079] The domain center vectors at each level are obtained by clustering the sample features of the corresponding level. The sample features are obtained by the backbone network extracting features from the samples in the first training set. The process of obtaining the domain center vectors is described below.

[0080] After training the backbone network based on the first dataset, multiple domain center vectors can be obtained through the trained backbone network. Optionally, the acquisition of domain center vectors can be completed before the image processing network is deployed to the application, or after the image processing network is deployed to the application, or at other times. This application does not limit this, as long as it ensures that the domain center vectors are obtained before the image to be processed is processed by the image processing network at the application.

[0081] In one possible implementation, when a backbone network is obtained by training the network based on a first training set, multiple sample features at each level can be obtained by processing the sample images in the first training set through multiple convolutional layers included in the backbone network; clustering can then be performed based on the multiple sample features at the same level to obtain multiple domain center vectors at each level.

[0082] For any number of sample features at any level, any clustering algorithm can be used to cluster these sample features to obtain multiple domain center vectors.

[0083] The clustering algorithm can be K-MEANS clustering algorithm, mean shift clustering algorithm, density-based spatial clustering of applications with noise (DBSCAN) algorithm, expectation-maximization (EM) clustering algorithm using Gaussian Mixture Model (GMM), hierarchical clustering algorithm, etc. This application does not limit which clustering algorithm is used.

[0084] It should be noted that for each level of convolutional layer, multiple corresponding domain center vectors can be obtained, so that processing can be performed based on the corresponding domain center vectors in each level of processing.

[0085] Specifically, when the feature vector corresponding to the first feature of the current level and the multiple domain center vectors of the current level are input into the first attention mechanism network of the current level, and the query vector of the current level is output through the first attention mechanism network of the current level, the first feature of the current level and the query vector can be weighted and summed through the first attention mechanism network of the current level to obtain the query vector of the current level.

[0086] The above explanation uses only one first attention mechanism network as an example. The processing of other first attention mechanism networks is similar and will not be repeated here.

[0087] Step 2: Based on the query vector of the current level and the third feature output by the previous level, the second feature of the current level is obtained through the second attention mechanism network of the current level.

[0088] Specifically, for the first level, the third feature output by the previous level is the first feature of the current level. For levels other than the first level, the third feature output by the previous level is obtained by feature fusion based on the first feature of the current level and the second feature of the previous level. In other words, when the current level is the first level of the image processing network, the third feature is the first feature of the first level; and when the current level is not the first level of the image processing network, the third feature is obtained by feature fusion based on the second feature of the previous level and the first feature of the current level.

[0089] The processing steps of the second attention mechanism module may include the following:

[0090] Step 1: Input the query vector of the current level and the third feature output from the previous level into the second attention mechanism network of the current level. Through the second attention mechanism network of the current level, the feature vectors of different dimensions in the query vector of the current level and the third feature output from the previous level are weighted. Based on the multiple vectors obtained after weighting, the first intermediate feature is determined.

[0091] It should be noted that before weighting the query vector at the current level and the third feature output from the previous level, the third feature output from the previous level can be split into multiple feature vectors with a dimension of 1. Then, the query vector and the multiple feature vectors with a dimension of 1 obtained from the splitting can be weighted separately to obtain the first intermediate feature.

[0092] Step 2: Perform convolution on the first intermediate feature and the third feature output from the previous level to obtain the second intermediate feature.

[0093] Step 3: Perform feature fusion processing based on the second intermediate feature and the third feature output from the previous level to obtain the second feature of the current level.

[0094] The above embodiments mainly introduce the process of obtaining the second feature of the current level. To facilitate understanding of the process of obtaining the second feature of each level, the following describes how to obtain the second feature of the next level after obtaining the second feature of the current level through steps one and two above. After performing steps one and two above, the second feature of the next level of the current level can also be obtained through steps three and four as follows:

[0095] Step 3: Perform feature fusion processing on the second feature of the current level and the first feature of the next level to obtain the third feature of the next level.

[0096] Step 4: Based on the third feature of the next level and the query vector of the next level, the second feature of the next level is obtained through the second attention mechanism network of the next level. The query vector of the next level is obtained based on the multiple domain center vectors of the next level and the first feature of the next level.

[0097] For ease of understanding, the following will be used as an example. Figure 2 Using the image processing network shown as an example, the feature processing process of the image processing method provided in this application will be described in detail.

[0098] See Figure 2 , Figure 2 This is a schematic diagram of the network structure of an image processing network according to an exemplary embodiment of this application. Figure 2 As shown, the image processing network may include a backbone network, a first attention mechanism network, a second attention mechanism network, a feature fusion network, and a classification network. The backbone network includes convolutional layers 211, 212, 213, and 214. The first attention mechanism network includes first attention mechanism networks 241, 242, 243, and 244. The second attention mechanism network includes second attention mechanism networks 231, 232, 233, and 234. The feature fusion network includes feature fusion units 221, 222, 223, and 224.

[0099] After acquiring the image to be processed, the image can be convolved through the first-level convolutional layer 211 to obtain the first feature A1 at the first level; then the first feature A1 can be convolved through the second-level convolutional layer 212 to obtain the first feature A2 at the second level; then the first feature A2 can be convolved through the third-level convolutional layer 213 to obtain the first feature A3 at the third level; finally, the first feature A3 can be convolved through the fourth-level convolutional layer 214 to obtain the first feature A4 at the fourth level. After obtaining the first features at each level, global pooling can be performed on the first feature A1 to obtain the feature vector corresponding to the first feature A1. The feature vector corresponding to the first feature A1, along with multiple domain center vectors of the first level, is then input into the first attention mechanism network 241 of the first level. The first attention mechanism network 241 outputs the query vector D1. The first feature A1 is then used as the third feature, and together with the query vector D1, it is input into the second attention mechanism network 231 of the first level. The second attention mechanism network 231 outputs the third feature B1 of the first level. Then, the third feature B1 and the first feature A2 are input into the feature fusion unit 221 to obtain the second feature C1. Furthermore, global pooling is performed on the first feature A2 to obtain the feature vector corresponding to the first feature A2. The feature vector corresponding to the first feature A2 and the multiple domain center vectors of the first feature A2 are then input into the first attention mechanism network 241. Multiple domain center vectors are input into the second-level first attention mechanism network 242. The first attention mechanism network 242 outputs query vector D2. Then, the second feature C1 and query vector D2 are input together into the second-level second attention mechanism network 232. The second attention mechanism network 232 outputs third feature B2. The third feature B2 and the first feature A3 are input into the feature fusion network 222 to obtain the second feature C2. This process continues until the second feature C3 and query vector D4 are finally input into the fourth-level second attention mechanism network 234. The second attention mechanism network 234 outputs third feature B4. The third feature B4 and the first feature A4 are input into the feature fusion network 224 to obtain the target feature. Based on the target feature, the classification network of the image processing network can output the classification label of the data to be processed.

[0100] In the second attention mechanism module, when processing features, the query vector and the third feature of the previous level are input into the attention mechanism layer. The attention mechanism layer performs weighted processing on the information of different dimensions in the query vector and the third feature of the previous level to output the first intermediate feature. Then, the first intermediate feature and the third feature of the previous level are convolved to obtain the second intermediate feature. In order to fuse the second intermediate feature and the third feature of the previous level, the second feature of the current level can be obtained.

[0101] Compared with related technologies that directly fine-tune all the parameters of the model (referred to as the fine-tuning scheme) to achieve model training, the fine-tuning scheme has a large dependence on downstream task data in terms of network training, while the scheme provided in this application achieves good prediction results with a small amount of data.

[0102] In contrast to related technologies that freeze the backbone network and only adjust the classifier parameters (referred to as the linear probing scheme), the linear probing scheme has a limited number of adjustable parameters in terms of network training, which limits the network's recognition accuracy. The recognition accuracy is far lower than that of the scheme provided in this application. In terms of network inference, the linear probing scheme requires training a new network for each client's request, which will bring huge overhead. In contrast, the scheme provided in this application can train multiple batch training tasks submitted by multiple users simultaneously.

[0103] Corresponding to the embodiments of the foregoing methods, this application also provides embodiments of the apparatus and the computing device on which it is applied.

[0104] like Figure 3 As shown, Figure 3 This is a block diagram of an image processing apparatus according to an exemplary embodiment of the present application, the apparatus comprising:

[0105] The feature acquisition module 301 is used to acquire first features of multiple levels of the image to be processed through the backbone network. The first features of the first level in the multiple levels are obtained by performing a convolution operation on the image features of the image to be processed, and the first features of non-first levels are obtained by performing a convolution operation on the first features of the previous level. The backbone network is trained based on the first training set.

[0106] The feature acquisition module 301 is also used to process multiple domain center vectors and the first feature of any level through an attention mechanism network to obtain the second feature. The attention mechanism network is trained based on the first training set. The domain center vectors of each level are obtained by clustering the sample features of the corresponding level. The sample features are obtained by the backbone network extracting features from the samples in the first training set.

[0107] The feature acquisition module 301 is also used to acquire target features based on the first and second features of the last level;

[0108] The result acquisition module 302 is used to obtain the classification result of the image to be processed based on the target features through the classification network, wherein the classification network is trained based on the second training set.

[0109] In some embodiments of this application, the attention mechanism network includes a first attention mechanism network and a second attention mechanism network;

[0110] The feature acquisition module 301, when used to process multiple domain center vectors and the first feature of the current level through an attention mechanism network to obtain the second feature for any level, is used for:

[0111] For any given level, based on the multiple domain center vectors of the current level and the first feature of the current level, the query vector is obtained through the first attention mechanism network of the current level.

[0112] Based on the query vector of the current level and the third feature output by the previous level, the second feature of the current level is obtained through the second attention mechanism network of the current level. For the first level, the third feature output by the previous level is the first feature of the current level. For non-first levels, the third feature output by the previous level is obtained by feature fusion processing based on the first feature of the current level and the second feature of the previous level.

[0113] In some embodiments of this application, the process of obtaining the query vector for any level includes:

[0114] Perform global pooling on the first feature of the current level to obtain the feature vector corresponding to the first feature of the current level;

[0115] The feature vector corresponding to the first feature of the current level and the multiple domain center vectors of the current level are input into the first attention mechanism network of the current level. The query vector of the current level is output through the first attention mechanism network of the current level.

[0116] In some embodiments of this application, the feature acquisition module 301, when inputting the feature vector corresponding to the first feature of the current level and multiple domain center vectors of the current level into the first attention mechanism network of the current level, and outputting the query vector of the current level through the first attention mechanism network of the current level, is used for:

[0117] The first attention mechanism network of the current level performs a weighted summation of the first feature of the current level and the query vector to obtain the query vector of the current level.

[0118] In some embodiments of this application, for any level of the second attention mechanism network, the processing procedure of the second attention mechanism network includes:

[0119] The query vector of the current level and the third feature output by the previous level are input into the second attention mechanism network of the current level. Through the second attention mechanism network of the current level, the feature vectors of different dimensions in the query vector of the current level and the third feature output by the previous level are weighted. The first intermediate feature is determined based on the multiple vectors obtained after weighting.

[0120] The first intermediate feature and the third feature output from the previous level are convolved to obtain the second intermediate feature;

[0121] The second intermediate feature and the third feature output from the previous level are used for feature fusion processing to obtain the second feature of the current level.

[0122] In some embodiments of this application, the feature acquisition module 301, when used to acquire target features based on the first and second features of the last level, is used to:

[0123] The target features are obtained by performing feature fusion processing based on the first feature of the last layer and the second feature of the last layer.

[0124] In some embodiments of this application, the backbone network includes multiple levels of convolutional layers;

[0125] Feature acquisition module 301, when used to acquire first features of multiple levels of the image to be processed through the backbone network, is used for:

[0126] For the first-level convolutional layer, the image features are convolved to obtain the first-level convolutional features, which are used as the first features of the first level.

[0127] For any convolutional layer other than the first level, the first feature output by the convolutional layer of the previous level is convolved by the convolutional layer of the current level to obtain the convolutional feature of the current level, which is used as the first feature of the current level, so as to obtain multiple first features of non-first levels.

[0128] In some embodiments of this application, the feature acquisition module 301 is further configured to, when a backbone network is obtained by training the network based on the first training set, process the sample images in the first training set through the multiple convolutional layers included in the backbone network to obtain multiple sample features at each level.

[0129] The device also includes:

[0130] Clustering module 303 is used to cluster multiple samples based on features at the same level to obtain multiple domain center vectors at each level.

[0131] In some embodiments of this application, the backbone network, multiple levels of attention mechanism networks, and classification network are encapsulated as an image processing network; wherein, the backbone network includes multiple convolutional layers, and the attention mechanism network includes multiple first attention mechanism networks and multiple second attention mechanism networks, and one convolutional layer of the backbone network corresponds to one first attention mechanism network and one second attention mechanism network.

[0132] An image processing apparatus is configured to, in response to acquiring an image to be processed, input the image to be processed into an image processing network, process the image to be processed through the image processing network to obtain a classification result of the image to be processed, and perform other computer vision tasks based on the classification result of the image to be processed.

[0133] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0134] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0135] This application also provides a computing device, see [link to relevant documentation] Figure 4 , Figure 4 This is a schematic diagram illustrating the structure of a computing device according to an exemplary embodiment of this application. Figure 4 As shown, the computing device includes a processor 410, a memory 420, and a network interface 430. The memory 420 stores computer instructions that can run on the processor 410. The processor 410 is used to implement the image processing method provided in any embodiment of this application when executing the computer instructions. The network interface 430 is used to implement input / output functions. In more possible implementations, the computing device may also include other hardware, which is not limited in this application.

[0136] This application also provides a computer-readable storage medium, which can take many forms, such as random access memory (RAM), volatile memory, non-volatile memory, flash memory, storage drive (e.g., hard disk drive), solid-state drive, any type of storage disk (e.g., optical disc, DVD, etc.), or similar storage media, or combinations thereof. Specifically, the computer-readable medium can also be paper or other suitable media capable of printing programs. A computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program implements the image processing method provided in any embodiment of this application.

[0137] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method provided in any embodiment of this application.

[0138] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, apparatus, computing device, computer-readable storage medium, or computer program product. Therefore, one or more embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification can take the form of a computer program product implemented on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-readable program code.

[0139] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments corresponding to computing devices are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0140] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of this application. In some cases, the actions or steps described in this application may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0141] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by an image processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0142] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0143] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0144] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0145] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0146] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0147] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of this application. In some cases, the actions described in this application may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0148] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. That is, this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

[0149] The above description is merely an optional embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification shall be included within the scope of protection of this specification.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a plurality of levels of first features of a to-be-processed image through a backbone network, wherein the first features of a first level in the plurality of levels are obtained by performing convolution operation on image features of the to-be-processed image, the first features of a non-first level are obtained by performing convolution operation on the first features of a previous level, the backbone network is trained based on a first training set, and the first training set comprises sample images of a plurality of business fields; for any level, performing attention mechanism processing on the plurality of domain center vectors and the first features of the current level through an attention mechanism network to obtain second features, wherein the attention mechanism network is trained based on the first training set, and the domain center vectors of each level are obtained by clustering sample features of the corresponding level, and the sample features are obtained by feature extraction on the samples in the first training set through the backbone network; obtaining target features according to the first features and the second features of the last level, and obtaining a classification result of the to-be-processed image based on the target features through a classification network, wherein the classification network is trained based on a second training set, and the sample images included in the second training set correspond to the same business field.

2. The method of claim 1, wherein, The attention mechanism network comprises a first attention mechanism network and a second attention mechanism network. The attention mechanism processing on the plurality of domain center vectors and the first features of the current level through the attention mechanism network to obtain the second features for any level comprises: for any level, obtaining a query vector based on the plurality of domain center vectors of the current level and the first features of the current level through the first attention mechanism network of the current level; obtaining the second features of the current level based on the query vector of the current level and the third features output by the previous level through the second attention mechanism network of the current level, wherein for the first level, the third features output by the previous level are the first features of the current level, and for the non-first level, the third features output by the previous level are obtained by performing feature fusion processing based on the first features of the current level and the second features of the previous level.

3. The method of claim 2, wherein, For the query vector of any level, the obtaining process of the query vector comprises: performing global pooling processing on the first features of the current level to obtain a feature vector corresponding to the first features of the current level; performing weighted summation processing on the first features of the current level and the plurality of domain center vectors through the first attention mechanism network of the current level to obtain the query vector of the current level.

4. The method according to claim 2 or 3, characterized in that, For the second attention mechanism network of any level, the processing process of the second attention mechanism network comprises: inputting the query vector of the current level and the third features output by the previous level into the second attention mechanism network of the current level, performing weighted processing on the feature vectors of different dimensions in the query vector of the current level and the third features output by the previous level through the second attention mechanism network of the current level, and determining a first intermediate feature based on a plurality of vectors obtained after the weighted processing; performing convolution processing on the first intermediate feature and the third features output by the previous level to obtain a second intermediate feature; and performing convolution processing on the second intermediate feature and the third features output by the previous level to obtain the second features of the current level. perform feature fusion processing based on the second intermediate feature and a third feature output by a previous level to obtain a second feature of a current level.

5. The method of claim 1, wherein, The target feature is obtained according to the first feature and the second feature of the last level, and includes: perform feature fusion processing based on the first feature of the last level and the second feature of the last level to obtain the target feature.

6. The method of claim 1, wherein, The backbone network includes a plurality of levels of convolutional layers. The first feature of each level of the image to be processed is obtained by the backbone network, and includes: For the first level of convolutional layer, the first level of convolutional feature is obtained by performing convolutional processing on the image feature by the first level of convolutional layer, as the first feature of the first level. For any convolutional layer other than the first level, the convolutional feature of the current level is obtained by performing convolutional processing on the first feature output by the convolutional layer of the previous level by the convolutional layer of the current level, as the first feature of the current level, to obtain the first feature of the plurality of levels other than the first level.

7. The method of claim 1, wherein, The method further includes: In the case of obtaining the backbone network by network training based on the first training set, a plurality of sample features of each level are obtained by processing the sample images in the first training set based on the plurality of convolutional layers included in the backbone network; A plurality of domain center vectors of each level are obtained by clustering the plurality of sample features of the same level respectively.

8. The method of claim 1, wherein, The backbone network, the plurality of levels of attention mechanism networks, and the classification network are encapsulated as an image processing network; wherein the backbone network includes a plurality of convolutional layers, the attention mechanism network includes a plurality of first attention mechanism networks and a plurality of second attention mechanism networks, and one convolutional layer of the backbone network corresponds to one first attention mechanism network and one second attention mechanism network; The method further includes: In response to obtaining the image to be processed, the image to be processed is input to the image processing network, and the image to be processed is processed by the image processing network to obtain a classification result of the image to be processed, so as to perform other computer vision tasks based on the classification result of the image to be processed.

9. An image processing apparatus characterized by comprising: The device includes: The feature acquisition module is configured to obtain a plurality of levels of first features of an image to be processed by a backbone network, wherein the first feature of the first level in the plurality of levels is obtained by performing convolutional operation on image features of the image to be processed, and the first feature of a level other than the first level is obtained by performing convolutional operation on the first feature of the previous level, the backbone network is trained based on a first training set, and the first training set includes sample images of a plurality of business domains; The feature acquisition module is further configured to, for any level, perform attention mechanism processing on the plurality of domain center vectors and the first feature of the current level by an attention mechanism network to obtain a second feature, wherein the attention mechanism network is trained based on the first training set, the domain center vectors of each level are obtained by clustering sample features of the corresponding level, and the sample features are obtained by feature extraction on the samples in the first training set by the backbone network. The feature acquisition module is further configured to, for any level, perform attention mechanism processing on the plurality of domain center vectors and the first feature of the current level by an attention mechanism network to obtain a second feature, wherein the attention mechanism network is trained based on the first training set, the domain center vectors of each level are obtained by clustering sample features of the corresponding level, and the sample features are obtained by feature extraction on the samples in the first training set by the backbone network. The feature acquisition module is further configured to acquire a target feature according to the first feature and the second feature of the last level. The result acquisition module is configured to acquire a classification result of the image to be processed based on the target feature through a classification network, wherein the classification network is trained based on a second training set, and sample images included in the second training set correspond to the same business field.

10. A computing device, comprising: The computing device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements operations performed by the image processing method according to any one of claims 1 to 8 when executing the program.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a program, and the program is executed by the processor to perform operations performed by the image processing method according to any one of claims 1 to 8.

Images