Training method, device, server and storage medium for image feature extraction model
By introducing a global attention module into the image feature extraction model, the correlation between parallel filters is enhanced, and the problem of low training efficiency of convolutional neural network model is solved, and more efficient feature extraction and training convergence is achieved.
Patent Information
- Application Number
- CN202111417594.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-26
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-26
AI Technical Summary
The training efficiency of image feature extraction based on convolutional neural network models is low, and the prior art is difficult to effectively improve the correlation and mutual perception between filters.
Using an image feature extraction model including a convolution module and a global attention module, convolution operations are performed through parallel filters, and the results are input into the global attention module to enhance the correlation between filters, and the parameters of the convolution module are adjusted to improve the Training efficiency.
By enhancing the correlation between filters, the efficiency of model training and feature expression capabilities are improved, the accuracy of image feature extraction and training convergence speed are improved.
Smart Images

Figure CN114119997B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computers, and in particular to a training method, device, server and storage medium for an image feature extraction model. Background Art
[0002] With the development of deep learning technology, the application of image feature extraction based on convolutional neural network models is becoming more and more widespread. The convolutional neural network model uses filters to filter and express features to extract image features.
[0003] However, to extract image features through a convolutional neural network model, the image must be input into the neural network model and then the neural network model parameters must be adjusted through iterative training, which results in low training efficiency. Summary of the invention
[0004] The embodiments of the present application provide a training method, device, server and storage medium for an image feature extraction model, which can enhance the correlation between filters and improve training efficiency.
[0005] The present application embodiment provides a method for training an image feature extraction model, including:
[0006] Obtaining image features of training samples and an image feature extraction model, wherein the image feature extraction model includes a convolution module and a global attention module, and the convolution module includes multiple parallel filters;
[0007] Through each of the parallel filters, a convolution operation is performed on the image features of the training sample to obtain image sub-features corresponding to each of the parallel filters;
[0008] Inputting all the image sub-features into a global attention module to obtain image features with global attention;
[0009] According to the image feature with global attention, adjusting the convolution module to obtain an adjusted convolution module;
[0010] According to the adjusted convolution module and the preset loss function, a trained image feature extraction model is obtained.
[0011] The present application also provides a training device for an image feature extraction model, including:
[0012] An acquisition unit, used to acquire image features of training samples and an image feature extraction model, wherein the image feature extraction model includes a convolution module and a global attention module, and the convolution module includes a combination filter and multiple parallel filters;
[0013] A convolution unit, configured to perform a convolution operation on the image features of the training sample through each of the parallel filters to obtain image sub-features corresponding to each of the parallel filters;
[0014] A global attention unit, used for inputting all the image sub-features into a global attention module to obtain image features with global attention;
[0015] An adjustment unit, configured to adjust the convolution module according to the image feature with global attention to obtain an adjusted convolution module;
[0016] The training unit obtains a trained image feature extraction model according to the adjusted convolution module and the preset loss function.
[0017] An embodiment of the present application also provides a server, including a memory storing multiple instructions; the processor loads instructions from the memory to execute the steps in any one of the image feature extraction model training methods provided in the embodiments of the present application.
[0018] An embodiment of the present application also provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the steps in any one of the image feature extraction model training methods provided in the embodiments of the present application.
[0019] The embodiment of the present application can obtain the image features of the training samples and the image feature extraction model, the image feature extraction model includes a convolution module and a global attention module, the convolution module includes multiple parallel filters; through each of the parallel filters, the image features of the training samples are convolved to obtain the image sub-features corresponding to each of the parallel filters; all the image sub-features are input into the global attention module to obtain the image features with global attention; according to the image features with global attention, the convolution module is adjusted to obtain the adjusted convolution module; according to the adjusted convolution module and the preset loss function, the trained image feature extraction model is obtained. In the present application, by inputting the convolution results of the parallel filters into the global attention module, attention is paid to the connection between the filters, so that each filter can perceive and influence each other during the training process, thereby improving the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0021] Figure 1a It is a scene schematic diagram of the training method of the image feature extraction model provided in the embodiment of the present application;
[0022] Figure 1b This is a first flow chart of a training method for an image feature extraction model provided in an embodiment of the present application;
[0023] Figure 2 This is a second flow chart of the training method of the image feature extraction model provided in the embodiment of the present application;
[0024] Figure 3 It is an architecture diagram of the image feature extraction model provided in the embodiment of the present application;
[0025] Figure 4 It is a flowchart of the Transformer module provided in the embodiment of the present application;
[0026] Figure 5 This is a schematic diagram of the first application process of the image feature extraction model provided in the embodiment of the present application;
[0027] Figure 6 This is a schematic diagram of a second application process of the image feature extraction model provided in an embodiment of the present application;
[0028] Figure 7 This is a schematic diagram of a third application process of the image feature extraction model provided in an embodiment of the present application;
[0029] Figure 8 It is a structural schematic diagram of a training device for an image feature extraction model provided in an embodiment of the present application;
[0030] Fig. 9 It is a schematic diagram of the structure of the server provided in the embodiment of the present application. DETAILED DESCRIPTION
[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0032] Embodiments of the present application provide a method, device, server, and storage medium for training an image feature extraction model.
[0033] Among them, the training device of the image feature extraction model can be specifically integrated in an electronic device, which can be a terminal, a server and other devices. Among them, the terminal includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. The server can be a single server or a server cluster composed of multiple servers. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.
[0034] In some embodiments, the training device of the image feature extraction model can also be integrated into multiple electronic devices. For example, the training device of the image feature extraction model can be integrated into multiple servers, and the training method of the image feature extraction model of the present application can be implemented by multiple servers.
[0035] In some embodiments, the server may also be implemented in the form of a terminal.
[0036] For example, refer to Figure 1a The server can be equipped with a training device for an image feature extraction model. The server can obtain image features of training samples and an image feature extraction model from a database. The image feature extraction model includes a convolution module and a global attention module. The convolution module includes multiple parallel filters. Through each parallel filter, a convolution operation is performed on the image features of the training sample to obtain image sub-features corresponding to each parallel filter. All image sub-features are input into the global attention module to obtain image features with global attention. According to the image features with global attention, the convolution module is adjusted to obtain an adjusted convolution module. According to the adjusted convolution module and a preset loss function, a trained image feature extraction model is obtained.
[0037] When a user uploads an image to be processed, the server can use the trained image feature extraction model to extract features from the image to be processed, obtain feature data of the image to be processed, and return the feature data of the image to be processed to the user.
[0038] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.
[0039] Artificial Intelligence (AI) is a technology that uses digital computers to simulate human perception of the environment, acquire knowledge, and use knowledge. This technology can enable machines to have functions similar to human perception, reasoning, and decision-making. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation, and other major directions.
[0040] Among them, machine learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.
[0041] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0042] In this embodiment, a training method for an image feature extraction model involving artificial intelligence is provided, such as Figure 1b As shown, the specific process of the training method of the image feature extraction model can be as follows:
[0043] 110. Obtain image features of training samples and an image feature extraction model, where the image feature extraction model includes a convolution module and a global attention module, and the convolution module includes multiple parallel filters.
[0044] The present application can be applied to various image processing scenarios based on artificial intelligence, such as image recognition, image classification, etc. When the trained image feature extraction model is obtained by training the image feature extraction model, the image can be input into the image feature extraction model for feature extraction, and image recognition or image classification is performed according to the extracted features. In particular, in the scenario of image classification, after obtaining the trained image feature extraction model, the image data to be classified can be input into the trained image feature extraction model, and after obtaining the feature data, the image classification is performed according to the feature data; the image data to be classified can also be input into the convolution layer in the prior art to obtain the feature data, and then the feature data is input into the trained image feature extraction model to obtain the new feature data, and the image classification is performed according to the new data features; the image data to be classified can also be input into the trained image feature extraction model and the convolution layer in the prior art at the same time to obtain two feature data, and the two feature data are merged to obtain new feature data, and the image classification is performed according to the new data features, etc. Among them, the method of image classification according to feature data can include but is not limited to using a fully connected network for image classification.
[0045] Among them, the training sample refers to the sample image required for training the neural network model, and in this application, it refers to the sample image required for training the image feature extraction model. The image features can be extracted from the sample image by at least one of SIFT (Scale-invariant feature transform) feature extraction method, HOG (Histogram of Oriented Gradient) feature extraction method, LBP (Local Binary Pattern) feature extraction method, convolutional neural network, residual neural network, etc. The image features can be represented in the form of feature maps.
[0046] Among them, the convolution module refers to a module that can perform convolution operations. The convolution module can be a convolution layer, which extracts different features of the input through convolution operations. The filter refers to the convolution unit in the convolution module, which can be a convolution kernel. When processing an image, the pixels in a small area of the input image are weighted averaged to become each corresponding pixel in the output image, and the function that defines the weight is the convolution kernel. The combined filter refers to a filter obtained by combining multiple filters. Multiple filters can be obtained by linearly combining multiple filters, or by nonlinearly combining multiple filters, for example, by nonlinear weighted averaging based on filters and filter weight factors, and so on. The parallel filter refers to a filter set in the convolution module for parallel convolution operations. Parallel filters and combined filters can also be dynamic convolution kernels, which realize dynamic changes in the parameters of the combined filter by learning the function that maps the input to the filter.
[0047] Among them, the global attention module refers to a module that can perform global attention calculations. It is a module built based on the attention mechanism. It can be an attention layer that can perform global attention calculations or a module containing an attention layer, for example, GSANet (Global Self-Attention Networks for Image Recognition), RNN (Recurrent Neural Network) with attention mechanism, and so on.
[0048] In some embodiments, in order to enhance the learning effect of the order relationship between sequences and improve the model training efficiency, the global attention module can be a Transformer module. The Transformer module is a model framework built on the basis of the attention mechanism, and its overall architecture can be divided into an input layer, an encoding layer, a decoding layer, and an output layer. For example, the global attention module can be a Transformer Encoder architecture, which is composed of a multi-head self-attention module and a feedforward network layer (MLP), and a LayerNorm operation is applied before each block, and a residual connection is applied after each block. If the Transformer module has multiple layers of decoding, the image feature matrix is input into the encoding layer, and the encoding layer performs self-attention calculation on the image feature matrix, and sends the image feature matrix after the self-attention calculation to the feedforward neural network of the layer, and sends the image feature matrix to the next layer of the encoding layer via the feedforward neural network of the layer, and repeats the corresponding processing until all the multiple layers of the encoding layer are executed, and then the residual block is used to decode the features, and finally a convolutional layer is used to obtain the result of feature extraction.
[0049] Before training the neural network model, the parameters are generally initialized randomly. In some embodiments, in order to improve the efficiency of model training, the weight parameters of the filter are obtained through the attention module before training the model. Specifically, before step 110, steps 1.1 to 1.3 may also be included, as follows:
[0050] 1.1. Obtain the initial convolution module;
[0051] 1.2. Input the image features of the training samples into the attention module to obtain the attention vector;
[0052] 1.3. Adjust the initial convolution module according to the attention vector to obtain the convolution module.
[0053] Among them, the attention module is a module that can perform attention calculations. It is a module built based on the attention mechanism. It can be an attention layer or a module containing an attention layer, for example, a CBAM module (Convolutional BlockAttention Module, lightweight attention module), a SE module (Squeeze-and-Excitation), and so on.
[0054] The attention vector is a vector used to adjust the filter weights in the initial convolution module. Each value of the attention vector is the weight corresponding to a filter in the initial convolution module. The dimension of the attention vector can be set according to actual needs.
[0055] Before obtaining the image feature extraction model, an initial image feature extraction model including an attention module, an initial convolution module, and a global attention module is constructed, and an attention vector is obtained through the attention module, so as to adjust the initial convolution module to a convolution module. The initial convolution module can be adjusted to a convolution module by, when the initial convolution module includes multiple initial filters, performing weight calculation on some or all of the initial filters based on the attention vector to obtain parallel filters, and performing nonlinear weighted averaging on some or all of the initial filters to obtain a combined filter.
[0056] In some embodiments, step 1.3 may include the following steps: in the attention module, average pooling is performed on the entire image of the image features of the training sample, and then normalized through two fully connected layers and a normalized exponential function (Softmax) to obtain an attention vector.
[0057] In some embodiments, the combined filter may be obtained by combining multiple parallel filters. The attention vector may also be linearly combined with multiple initial filters to obtain a combined filter.
[0058] In some embodiments, in order to increase the relevance of parallel filters and combined filters, the convergence speed of the model training process is improved to improve the model training efficiency. Specifically, the initial convolution module includes multiple initial filters, the attention vector includes multiple values corresponding to the multiple initial filters, and step 1.3 may include steps 1.3.1 to 1.3.2, as follows:
[0059] 1.3.1. For each initial filter, multiply the initial filter by the corresponding value in the attention vector to obtain a parallel filter corresponding to each initial filter;
[0060] 1.3.2. Based on the parallel filter, a convolution module is obtained.
[0061] In some implementations, before step 110, a step may also be included: extracting features from the image data input of the training sample to obtain image features of the training sample.
[0062] 120. Through each parallel filter, a convolution operation is performed on the image features of the training sample to obtain image sub-features corresponding to each parallel filter.
[0063] The image sub-features refer to image features that characterize local features of an image.
[0064] By inputting the image features of the training sample into any parallel filter, the expression result of the parallel filter, ie, the image sub-feature, is obtained. In step 120, the above operations are performed on all parallel filters at the same time, so that a plurality of image sub-features corresponding to all parallel filters can be obtained.
[0065] 130. All image sub-features are input into the global attention module to obtain image features with global attention.
[0066] Among them, the image features with global attention refer to the image features obtained by the global attention module performing global attention calculation on all image sub-features. For example, in the RNN module with attention mechanism, the global attention can be calculated by the attention scoring mechanism. For example, at each hidden state step output by the RNN module, the historical state and hidden state scores can be calculated according to the attention scoring mechanism, and then the global attention for the hidden state can be generated through the normalized exponential function to obtain the global attention.
[0067] By inputting image sub-features into the global attention module, attention is paid to the connection between image sub-features. Since image sub-features are obtained one by one by parallel filters, by paying attention to the connection between image sub-features, the connection between multiple parallel filters is reflected.
[0068] In some implementations, in order to obtain feature data that can be used to characterize the entire image of the training sample and enhance the image feature expression effect, step 130 may include steps 2.1 to 2.2, as follows:
[0069] 2.1. Input all image sub-features into the global attention module, and for each image sub-feature, obtain the image sub-feature with global attention corresponding to the image sub-feature;
[0070] 2.2. Merge all image sub-features with global attention to obtain image features with global attention.
[0071] By performing global attention calculation on any image sub-feature in the global attention module, the image sub-feature with global attention corresponding to the image sub-feature is obtained. In step 2.1, this operation is performed on all image sub-features, so that multiple image sub-features with global attention corresponding to all image sub-features can be obtained. All the image sub-features with global attention are then merged together in the channel dimension to obtain an image feature with global attention, which can be used to characterize the features of the entire image.
[0072] In some embodiments, when training the image feature map of the sample, the total image features, image sub-features, image sub-features with global attention, and image features with global attention are also feature maps. After obtaining the image features with global attention in step 2.2, the step may also be included: adjusting the feature map size of the image features with global attention through a convolution operation so that the feature map size of the image features with global attention is consistent with that of the total image features.
[0073] In some embodiments, when training the image feature map of the sample, before step 2.1, a step of transforming each image sub-feature into a vector by a Reshape function may be included. Between steps 2.1 and 2.2, a step of transforming the image sub-feature with global attention by a Reshape function may be included so that the image sub-feature with global attention has the same feature map size as the image sub-feature. Among them, the Reshape function is a function that transforms a specified matrix into a matrix of a specific dimension, and the number of elements in the matrix remains unchanged. The function can readjust the number of rows, columns, and dimensions of the matrix.
[0074] In some embodiments, in order to enhance the connection between the parallel filters in the convolution module and enable the parallel filters to perceive and influence each other during the training process to improve the training efficiency, a global attention calculation can be performed based on the relationship values of the image sub-features calculated by the parallel filters. Specifically, step 2.1 can include steps 2.2.1 to 2.2.5, as follows:
[0075] 2.2.1. Input all image sub-features into the global attention module;
[0076] 2.2.2. For each image sub-feature, determine the image sub-feature as a first image sub-feature, and determine all image sub-features except the first image sub-feature as second image sub-features;
[0077] 2.2.3. Determine a first relationship value, a second relationship value, and a feature value corresponding to the first image sub-feature, and a first relationship value, a second relationship value, and a feature value corresponding to the second image sub-feature;
[0078] 2.2.4. Perform target operations on the first relationship value corresponding to the first image sub-feature and the second relationship value and feature value corresponding to each second image sub-feature to obtain multiple operation results;
[0079] 2.2.5. Add multiple operation results to obtain an image sub-feature with global attention corresponding to the first image sub-feature.
[0080] Wherein, when the image sub-feature is the first image sub-feature, the first relationship value may be a value used to learn the relationship between the first image sub-feature and the second image sub-feature, the second relationship value may be a value used to learn the relationship between the second image sub-feature and the first image sub-feature, the characteristic value may be specific information representing each first image sub-feature, the first relationship value, the second relationship value and the characteristic value may be set according to actual needs, or may be a parameter value to be learned determined by the attention mechanism, the first relationship value may be a Q (query) value, the second relationship value may be a K (key) value, and the characteristic value may be a V (value) value. The Q value, the K value and the V value are values corresponding to the first image sub-feature in the Q, K and V matrices, respectively. For example, Q, K and V may be obtained by different linear transformations of the embedding results.
[0081] The target operation may be a linear operation, such as addition, multiplication, etc., or a nonlinear operation, such as logarithmic operation, square root operation, exponential operation, trigonometric function operation, etc.
[0082] For example, when the global attention module is a Transformer module, the global attention can be calculated by K (key), Q (query) and V (value), where Q is used to learn the relationship between itself and other elements, K is used to learn the relationship between other elements and itself, and V represents the specific information of each element, and the elements correspond to the image sub-features of the embodiment of the present application. The target operation can be a product operation, such as multiplying Km, Q, and V in sequence to obtain a result with neighborhood information, that is, an image sub-feature with global attention. Specifically, in the process of performing the target operation, Q is a matrix composed of the first relationship values of all image sub-features in sequence, K m is the relationship value of the first image sub-feature, m represents the sequence number of the first image sub-feature in the series, the sequence can be determined according to the sorting method of the parallel convolution kernel, or can be set as needed, K is a matrix composed of the second relationship values of all image sub-features in sequence, and V is a matrix composed of the eigenvalues of all image sub-features in sequence. Assume X∈R n×dis an input sample sequence, a sample is an image sub-feature, and a sample sequence is all input image sub-features, where n is the number of samples (sequence length) and d is the latitude of a single sample. Query, Key, and Value are defined as: Query: Q = X × W Q , where W Q ∈R d×dq ,Key:K=X×W K , where W K ∈R d×dk , Value: V = X × W V , where W K ∈R d×dv .
[0083] 140. According to the image features with global attention, the convolution module is adjusted to obtain an adjusted convolution module.
[0084] In some embodiments, the convolution module further includes a combined filter, and step 140 may include steps 3.1 to 3.2, as follows:
[0085] 3.1. By combining filters, convolution operation is performed on the image features of the training samples to obtain the total image features;
[0086] 3.2. According to the image features with global attention and the total image features, the convolution module is adjusted to obtain an adjusted convolution module.
[0087] In some embodiments, the attention vector is linearly combined with the plurality of initial filters to obtain a combined filter. The step of obtaining a convolution module according to all parallel filters may be: obtaining a convolution module according to the combined filter and all parallel filters.
[0088] The linear combination refers to linearly weighting multiple initial filters according to the attention vector. By performing two calculation processes on the attention vector and multiple initial filters, a combined filter and a parallel filter are obtained. In the convolution module, two groups of parallel convolution units can perform convolution operations in parallel, where one group of convolution units includes multiple parallel filters and the other group of convolution units includes a combined filter.
[0089] Among them, the total image feature refers to the image feature that can characterize the features of the entire image. The total image feature extracted by the combined filter has more receptive fields. Since the combined filter aggregates multiple filters, it has a larger receptive field than a single filter, so it has greater feature expression ability, especially compared with the parallel combiner, which can extract image features with more receptive fields. The receptive field of the total image feature can have a mapping relationship with the image features of the training sample.
[0090] In some embodiments, the combined filter includes a BN (Batch Normalization) layer and an activation layer, in which the image features of the training sample are passed through the BN layer and the activation layer to obtain the total image features. The activation function of the activation layer may be a ReLU function, a Swish function, a GLU activation function, and the like.
[0091] By using a combination of filters to extract the image features of the training samples, the total image features are obtained, and the total image features are used as the standard value to perform supervision tasks during the training process. The supervision information gradient is fed back to influence the filter and the attention vector learning, ensuring effective learning between filters and enhancing the expression effect of image features.
[0092] In some embodiments, step 3.2 may include steps 3.2.1 to 3.2.2, as follows:
[0093] 3.2.1. According to the total features of the image and the features of the image with global attention, gradient reversal is performed to obtain the gradient reversal result;
[0094] 3.1.2. According to the gradient inversion result, adjust the convolution module to obtain the adjusted convolution module.
[0095] Among them, gradient reversal is to pass the loss (the difference between the predicted value and the true value) backward layer by layer, and then each layer of the network will calculate the gradient according to the error passed back, and then update the parameters of the network at this layer. In the embodiment of the present application, the image features with global attention are passed back layer by layer to the convolution module, and the convolution module calculates the gradient according to the loss to update the parameters in the convolution module.
[0096] In some embodiments, based on the output image features with global attention, the parameters of the filters in the convolution module can be adjusted to enhance the connection between the parallel filters in the convolution module and improve the training efficiency. Adjusting the convolution module refers to adjusting the parameters of the filters and the attention vector, for example, the convolution module can be adjusted by an adaptive algorithm or a gradient reversal algorithm.
[0097] In some embodiments, since the parallel filter and the combined filter are obtained by the initial filter, the parallel filter and the combined filter can be adjusted simultaneously by the total image features and the image features with global attention, so that the obtained parallel filter and the combined filter can learn the correlation between different filters and enhance the image feature expression effect. Specifically, step 140 may include the steps of: adjusting the parallel filter and the combined filter according to the total image features to obtain the adjusted convolution module. For example, the initial filter parameters and corresponding weights are adjusted by the total image features and the image features with global attention. Since the parallel filter and the combined filter are calculated by the initial filter, the parallel filter and the combined filter are adjusted by adjusting the initial filter.
[0098] 150. According to the adjusted convolution module and the preset loss function, a trained image feature extraction model is obtained.
[0099] Among them, the preset loss function refers to a function that is preset to evaluate whether the model training is completed. For example, in supervised learning, the preset loss function can be used to evaluate the degree to which the predicted value of the model is different from the true value. It can be any loss function, such as log loss function, square loss function, exponential loss function or Hinge loss function, etc., which can be set according to specific needs. For example, the total image features can be used as the true value, and the image features with global attention can be used as the predicted value, and the loss function value can be calculated according to the preset loss function. Or the preset loss can be used to evaluate whether the image feature extraction model converges.
[0100] After obtaining the total image features and the image features with global attention, the total image features can be used for supervised learning. When the total image features and the image features with global attention converge, a trained image feature extraction model is obtained. It should be noted that in the process of training the neural network model, the model parameters will be continuously adjusted to make the model converge to obtain the trained model.
[0101] In some implementations, the preset loss function may be MSE Loss (mean square error loss function).
[0102] In some embodiments, after obtaining the trained image feature extraction model, when inputting the image data to be processed into the trained image feature extraction model, steps 4.1 to 4.2 may be included as follows:
[0103] 4.1. Input the image data to be processed into the trained image feature extraction model;
[0104] 4.2. Through the convolution module, a convolution operation is performed on the image data to be processed to obtain the feature data of the image to be processed.
[0105] The image data to be processed may be image data that has not been extracted, or image features that have been extracted. It should be noted that the image feature extraction model trained in the present application may be used to replace the filter in the neural network used to extract image features in the prior art.
[0106] In some embodiments, when adjusting the parallel filter parameters of the convolution module, step 4.2 may include the steps of: performing a convolution operation on the image data to be processed through a parallel filter to obtain multiple sub-features of the image data to be processed; inputting the multiple sub-features into the global attention module to obtain feature data of the image to be processed.
[0107] In some embodiments, when adjusting the combined filter parameters of the convolution module, step 4.2 may include the step of: performing a convolution operation on the image data to be processed by combining the filters to obtain feature data of the image to be processed.
[0108] In some embodiments, when the parallel filter parameters and the combined filter parameters of the convolution module are adjusted simultaneously, step 4.2 can be any of the above two methods. In some embodiments, in order to improve the efficiency of data extraction, step 4.2 is: through the combined filter, the image data to be processed is convolved to obtain the feature data of the image to be processed.
[0109] In some embodiments, step 150 may include the steps of: based on the adjusted convolution module, according to a preset loss function, converging the total image features and the image features with global attention to obtain a trained image feature extraction model.
[0110] In some embodiments, feature data of the image to be processed is obtained based on the total image features, image features with global attention, and a preset loss function, including: converging the total image features and the image features with global attention according to a preset loss function; when the total image features and the image features with global attention converge, the feature data of the image to be processed is obtained.
[0111] In some embodiments, the preset loss function includes a first loss function and a second loss function, the first loss function is used to converge the total image features and the image features with global attention, and the second loss function is used to converge the image feature extraction model. The preset loss function in step 3.2 is the first loss function, and step 150 may also include: the trained image feature extraction model satisfies the preset model convergence condition. In some embodiments, the model convergence condition may be that the second loss function is less than a preset value and reaches a maximum number of iterations.
[0112] The training scheme of the image feature extraction model provided in the embodiment of the present application can be applied to various scenarios of image processing based on artificial intelligence. For example, taking image classification as an example, the image features of the training samples and the image feature extraction model are obtained, and the image feature extraction model includes a convolution module and a global attention module, and the convolution module includes multiple parallel filters; through each parallel filter, the image features of the training samples are convolved to obtain the image sub-features corresponding to each parallel filter; all the image sub-features are input into the global attention module to obtain the image features with global attention; according to the image features with global attention, the convolution module is adjusted to obtain the adjusted convolution module, and then the image data to be classified is input into the trained image feature extraction model, and after obtaining the feature data, the image is classified according to the feature data.
[0113] The solution provided by the embodiment of the present application sets parallel filters and combined filters in the convolution module, takes the convolution result of the combined filter as the standard value, and the convolution result of the parallel filter as the predicted value, so as to perform supervision tasks in the model training process, and influence the filter and attention vector learning through the supervision information gradient feedback, so as to ensure effective learning between filters. Since each filter constrains each other in the learning process, each filter is more targeted, the expression ability of the model is stronger, and the expression effect of image features can be enhanced. Furthermore, by inputting the convolution result of the parallel filter into the global attention module, paying attention to the connection between the filters to adjust the convolution filter parameters, the filters can be made to perceive and influence each other during the training process, and the relationship between different filters can be learned, the learning process is more direct, and the training convergence speed is accelerated.
[0114] Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0115] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operational detection. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining public and private key generation (account management), key management, and the maintenance of the correspondence between the user's real identity and the blockchain address (authority management), etc., and, under authorization, supervises and audits the transactions of certain real identities and provides risk control rule configuration (risk control audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests, and records valid requests to storage after consensus is reached. For a new business request, the basic service first performs interface adaptation analysis and authentication processing (interface adaptation), and then encrypts the business information through the consensus algorithm (consensus management). The smart contract module is responsible for the registration and issuance of contracts, as well as contract triggering and contract execution. Developers can define the contract logic in a programming language and publish it to the blockchain (contract registration). According to the logic of the contract terms, the key or other events are called to trigger the execution and complete the contract logic. It also provides the function of contract upgrade and cancellation. The operation detection module is mainly responsible for the deployment, configuration modification, contract setting, cloud adaptation and real-time status visualization output of the product during the product release process, such as alarm, network status detection, node equipment health status detection, etc.
[0116] The platform product service layer provides the basic capabilities and implementation framework of typical applications. Developers can superimpose business features based on these basic capabilities to complete the blockchain implementation of business logic. The application service layer provides application services based on blockchain solutions for business participants to use.
[0117] In one embodiment, the server provided by the present application can be used as a node in the blockchain system. After obtaining the trained image feature extraction model, the image to be processed is input into the image feature extraction model. After obtaining the image features of the image to be processed, the image features of the image to be processed are verified. After the verification is passed, the image features are stored in the blockchain as a new block to ensure that these extraction results will not be tampered with. When the trained image feature extraction model is applied to image analysis, the image analysis results can also be verified. After the verification is passed, the image analysis results can be stored in the blockchain as a new block to ensure that these extraction results will not be tampered with.
[0118] The method described in the above embodiment will be further described in detail below.
[0119] In this embodiment, image classification is taken as an example to illustrate the method of the embodiment of the present application in detail.
[0120] like Figure 2 As shown, the specific process of a training method for an image feature extraction model is as follows:
[0121] 210. Input the feature map of the training sample into the attention module, perform average pooling on the entire feature map of the training sample, and then pass it through two fully connected layers and a normalized exponential function for normalization to obtain an attention vector.
[0122] like Figure 3 As shown in FIG. 1 , a schematic diagram of the image feature extraction model of the present embodiment is provided. A feature map is input to the image feature extraction model, that is, Figure 3 After x in the attention module is input, the feature map is averaged and pooled (Avg Pool) is performed on the whole image in the attention module, and then normalized by two fully connected layers (FC) and Softmax function to obtain a K-dimensional vector. Each value in this vector is the convolution kernel Conv1 to Conv k The corresponding weight of . A ReLU activation function is connected between the two fully connected layers (FC).
[0123] 220. Linearly combine the attention vector with multiple initial convolution kernels to obtain a combined convolution kernel.
[0124] like Figure 3 As shown, the K-dimensional vector and Conv1 to Conv k Perform linear combination to obtain the combined convolution kernel Conv.
[0125] 230. The feature map of the training sample is input into the combined convolution kernel, passes through the BN layer and the activation layer, and outputs the total feature map.
[0126] like Figure 3 As shown, x is input into Conv to get the output Featuremap m The input passes through Conv, BN (Batch Normalization) layer and activation layer, and directly performs subsequent supervision tasks. The supervision information gradient is passed back to affect Conv, K-dimensional vector and Conv1 to Conv k of learning.
[0127] 240. For each initial convolution kernel, multiply the initial convolution kernel by the corresponding value in the attention vector to obtain a parallel convolution kernel corresponding to each initial convolution kernel.
[0128] 250. Through each parallel convolution kernel, the image features of the training sample are convolved to obtain a sub-feature map corresponding to each parallel convolution kernel.
[0129] like Figure 3 As shown, x is input to Conv n and The convolution kernel Conv1 to Conv after multiplication k (Among them, is an element in the K-dimensional vector, and n is any integer from 1 to k), and different expressions of each convolution kernel are obtained, namely Featuremap1 to Featuremap k .
[0130] 260. All sub-feature maps are input into the Transformer module. For each sub-feature map, a sub-feature map with global attention corresponding to the image sub-feature map is obtained. All sub-feature maps with global attention are merged to obtain a feature map with global attention.
[0131] like Figure 3 As shown, Featuremap1 to Featuremap k The input is sent to the Transformer module to learn the relationship between the results of each convolution kernel expression, and then the Transformer module output is connected with Featuremap1 to Featuremap k Featuremap of the same size but with contact information t1 To Featuremap tk , through average pooling (Avg Pool), all featuremaps are merged together in the channel dimension to obtain featuremap t , use 1×1 convolution to get the size and Featuremap m Consistent Featuremap ’ t .
[0132] Specifically, each Featuremap is an element in the sequence input. Input Featuremap n (n is any integer from 1 to k), the network can learn the corresponding K n (key), Q n (query) and V n (value), where Q is used to learn the relationship between itself and other sequence elements, K is used to learn the relationship between other sequence elements and itself, and V represents the specific information of each element.
[0133] For example, for Featuremap1, if the Transformer module is used to learn the expression information between neighbors, such as Figure 4As shown, the following steps are included: 1. Use the Reshape function to straighten Featuremap1 into a vector. Each vector is converted into Q1, K1, and V1 through Transformer. Other Featuremaps can also obtain corresponding Q, K, and V. 2. Connect Q1 and the corresponding K n Matrix multiplication, get the corresponding a 1,n , a 1,n and the corresponding V n Matrix multiplication is performed to obtain the intermediate representation of the nth sequence element. 3. Add the K intermediate representation results to obtain the output Featuremap with neighborhood information of Featuremap1 t1 . Featuremap t2 To Featuremap tk 4. Use the Reshape function to get the Featuremap tn (n is any integer from 1 to k) Restore to Featuremap t Consistent size.
[0134] 270. Perform gradient inversion on the image features with global attention to obtain a gradient inversion result, and adjust the initial convolution kernel and the attention vector according to the gradient inversion result.
[0135] like Figure 3 As shown, MSE Loss (Mean Squared Error Loss) is used to force Featuremap in learning. m With Featuremap ’ t Close, through gradient back propagation, so that the K-dimensional vector and Conv1 to Conv k Learn the connections between different convolution kernel elements.
[0136] 280. Loop through steps 220 to 280 until both the first loss function and the second loss function converge to obtain a trained image feature extraction model. The first loss function is used to converge the total feature map and the image features with global attention, and the second loss function is used to converge the image feature extraction model.
[0137] 290. Use the trained image feature extraction model for image classification.
[0138] This technology is used in image processing such as Figures 5 to 7 As shown, specific application methods include but are not limited to Figures 5 to 7 The trained image feature extraction model of the embodiment of the present application can replace the daily convolution kernel, and the input and output are the same as the daily convolution kernel input and output.
[0139] Specifically, Figure 5 As shown in , the image to be classified is input, and the image is extracted through the trained image feature extraction model to obtain a feature map, and the image is classified according to the obtained feature map. Figure 6 As shown, an image to be classified is input, a feature map is obtained through the convolution layer in the prior art, and then the feature map is input into the trained image feature extraction model to obtain a new feature map, and the image is classified according to the new feature map, wherein the number of layers used in the trained image feature extraction model and the order of the layers with the ordinary convolution layer are not limited. Figure 7 As shown, the image to be classified is simultaneously input into the trained image feature extraction model and the convolution layer in the prior art to obtain two feature maps. The two feature maps are merged to obtain a new feature map, and then the image is classified according to the new feature map.
[0140] It should be noted that the image feature extraction model trained in this application can also be used in language processing technology.
[0141] It can be seen from the above that the training method of the image feature extraction model of the embodiment of the present application can be used in different image classification methods. By learning the connection between different convolution kernels according to the K, Q, and V values in the Transformer module, the convolution kernels are connected in learning, so that each convolution kernel element can perceive and influence each other during the training process, and the learning process is more direct, the training convergence speed is accelerated, and the training efficiency can be improved. At the same time, since each convolution kernel element constrains each other during the learning process, each convolution kernel element is more targeted and the model has stronger expression ability. In addition, the attention vector is combined with the convolution kernel to obtain a combined convolution kernel and multiple parallel convolution kernels. The output result of the combined convolution kernel is used to supervise the training process. The convolution kernel and attention vector learning are affected by the supervision information gradient feedback, so as to ensure effective learning between filters and enhance the image feature expression effect.
[0142] In order to better implement the above method, the embodiment of the present application also provides a training device for an image feature extraction model, which can be integrated in an electronic device, which can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.
[0143] For example, in this embodiment, the method of the embodiment of the present application is described in detail by taking the training device of the image feature extraction model specifically integrated in the server as an example.
[0144] For example, Figure 8As shown, the training device of the image feature extraction model may include an acquisition unit 310, a convolution unit 320, a global attention unit 330, an adjustment unit 340 and a training unit 350, as follows:
[0145] (I) Acquisition Unit 310
[0146] The image features of the training samples and the image feature extraction model are obtained. The image feature extraction model includes a convolution module and a global attention module. The convolution module includes multiple parallel filters.
[0147] In some embodiments, the image feature extraction model further includes an attention module, and the acquisition unit may also be used to:
[0148] Get the initial convolution module;
[0149] Input the image features of the training samples into the attention module to obtain the attention vector;
[0150] The initial convolutional module is adjusted according to the attention vector to obtain the convolutional module.
[0151] In some embodiments, the initial convolution module includes a plurality of initial filters, the attention vector includes a plurality of values corresponding to the plurality of initial filters one by one, and the initial convolution module is adjusted according to the attention vector to obtain the convolution module, including:
[0152] For each initial filter, multiply the initial filter by the corresponding value in the attention vector to obtain a parallel filter corresponding to each initial filter;
[0153] According to all parallel filters, a convolution module is obtained.
[0154] (ii) Convolutional unit 320
[0155] It is used to perform convolution operation on the image features of the training samples through each parallel filter to obtain the image sub-features corresponding to each parallel filter.
[0156] (IV) Global Attention Unit 330
[0157] Used to input all image sub-features into the global attention module to obtain image features with global attention.
[0158] In some implementations, the global attention unit 330 may be specifically used to:
[0159] All image sub-features are input into the global attention module, and for each image sub-feature, an image sub-feature with global attention corresponding to the image sub-feature is obtained;
[0160] All image sub-features with global attention are merged to obtain image features with global attention.
[0161] In some implementations, all image sub-features are input into a global attention module, and for each image sub-feature, an image sub-feature with global attention corresponding to the image sub-feature is obtained, including:
[0162] Feed all image sub-features into the global attention module;
[0163] For each image sub-feature, determining the image sub-feature as a first image sub-feature, and determining all image sub-features except the first image sub-feature as second image sub-features;
[0164] Determine a first relationship value, a second relationship value, and a feature value corresponding to the first image sub-feature, and a first relationship value, a second relationship value, and a feature value corresponding to the second image sub-feature;
[0165] Performing target operations on the first relationship value corresponding to the first image sub-feature and the second relationship value and feature value corresponding to each second image sub-feature to obtain multiple operation results;
[0166] The multiple operation results are added together to obtain an image sub-feature with global attention corresponding to the first image sub-feature.
[0167] (IV) Adjustment unit 340
[0168] Used to adjust the convolution module according to the image features with global attention, and obtain the adjusted convolution module.
[0169] In some implementations, the convolution module further includes a combined filter, and the adjustment unit 340 may be specifically configured to:
[0170] By combining filters, the image features of the training samples are convolved to obtain the total image features;
[0171] According to the image features with global attention and the total image features, the convolution module is adjusted to obtain an adjusted convolution module.
[0172] In some implementations, adjusting the convolution module according to the image features with global attention and the total image features to obtain an adjusted convolution module includes:
[0173] According to the total features of the image and the image features with global attention, gradient reversal is performed to obtain the gradient reversal result;
[0174] According to the gradient inversion result, the convolution module is adjusted to obtain an adjusted convolution module.
[0175] (V) Training Unit 350
[0176] Used to obtain a trained image feature extraction model based on the adjusted convolution module and the preset loss function.
[0177] In some implementations, the training unit 350 may be specifically configured to:
[0178] Adjust the convolution module according to the total image features and the image features with global attention to obtain an adjusted convolution module;
[0179] Based on the adjusted convolution module, according to the preset loss function, the total image features and image features with global attention are converged to obtain the trained image feature extraction model.
[0180] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.
[0181] As can be seen from the above, the training device of the image feature extraction model of this embodiment may include an acquisition unit, which is used to acquire the image features of the training samples and the image feature extraction model, the image feature extraction model includes a convolution module and a global attention module, the convolution module includes a combined filter and a plurality of parallel filters; a convolution unit, which is used to perform a convolution operation on the image features of the training samples through each parallel filter to obtain the image sub-features corresponding to each parallel filter; a global attention unit, which is used to input all image sub-features into the global attention module to obtain image features with global attention; an adjustment unit, which is used to adjust the convolution module according to the image features with global attention to obtain an adjusted convolution module; a training unit, which obtains a trained image feature extraction model according to the adjusted convolution module and a preset loss function.
[0182] Therefore, the embodiment of the present application can input the convolution results of the parallel filters into the global attention module, pay attention to the connection between the filters, and dynamically adjust the convolution filter parameters so that each filter can perceive and influence each other during the training process, thereby improving the training efficiency.
[0183] The embodiment of the present application also provides an electronic device, which can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc. The server can be a single server or a server cluster composed of multiple servers, etc.
[0184] In some embodiments, the electronic device apparatus may also be integrated into multiple electronic devices. For example, the electronic device apparatus may be integrated into multiple servers, and the training method of the image feature extraction model of the present application may be implemented by multiple servers.
[0185] In this embodiment, the electronic device of this embodiment is a server as an example for detailed description, for example, Fig. 9 As shown, it shows a schematic diagram of the structure of the server involved in the embodiment of the present application, specifically:
[0186] The server may include one or more processing core processors 410, one or more computer-readable storage media memories 420, a power supply 430, an input module 440, and a communication module 450. Those skilled in the art will appreciate that Fig. 9 The server structure shown in the figure does not constitute a limitation on the server, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:
[0187] The processor 410 is the control center of the server. It uses various interfaces and lines to connect various parts of the entire server. It executes various functions of the server and processes data by running or executing software programs and / or modules stored in the memory 420, and calling data stored in the memory 420, so as to perform overall detection of the server. In some embodiments, the processor 410 may include one or more processing cores; in some embodiments, the processor 410 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 410.
[0188] The memory 420 can be used to store software programs and modules. The processor 410 executes various functional applications and data processing by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 420 may also include a memory controller to provide the processor 410 with access to the memory 420.
[0189] The server also includes a power supply 430 for supplying power to various components. In some embodiments, the power supply 430 can be logically connected to the processor 410 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 430 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.
[0190] The server may further include an input module 440, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.
[0191] The server may further include a communication module 450. In some embodiments, the communication module 450 may include a wireless module. The server may perform short-range wireless transmission through the wireless module of the communication module 450, thereby providing wireless broadband Internet access to the user. For example, the communication module 450 may be used to help the user send and receive emails, browse web pages, and access streaming media.
[0192] Although not shown, the server may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 410 in the server will load the executable files corresponding to the processes of one or more application programs into the memory 420 according to the following instructions, and the processor 410 will run the application programs stored in the memory 420, thereby realizing various functions, as follows:
[0193] Obtaining image features of training samples and an image feature extraction model, the image feature extraction model includes a convolution module and a global attention module, and the convolution module includes multiple parallel filters;
[0194] Through each parallel filter, the image features of the training sample are convolved to obtain the image sub-features corresponding to each parallel filter;
[0195] All image sub-features are input into the global attention module to obtain image features with global attention;
[0196] According to the image features with global attention, the convolution module is adjusted to obtain an adjusted convolution module.
[0197] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0198] From the above, we can see that the server inputs the convolution results of parallel filters into the global attention module, pays attention to the connection between filters, and dynamically adjusts the convolution filter parameters so that each filter can perceive and influence each other during the training process, thereby improving training efficiency.
[0199] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0200] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the training methods for image feature extraction models provided in the embodiments of the present application. For example, the instructions can execute the following steps:
[0201] Obtaining image features of training samples and an image feature extraction model, the image feature extraction model includes a convolution module and a global attention module, and the convolution module includes multiple parallel filters;
[0202] Through each parallel filter, the image features of the training sample are convolved to obtain the image sub-features corresponding to each parallel filter;
[0203] All image sub-features are input into the global attention module to obtain image features with global attention;
[0204] According to the image features with global attention, the convolution module is adjusted to obtain an adjusted convolution module.
[0205] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0206] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any one of the training methods for the image feature extraction model provided in the above embodiments.
[0207] Since the instructions stored in the storage medium can execute the steps in the training method of any image feature extraction model provided in the embodiments of the present application, the beneficial effects that can be achieved by the training method of any image feature extraction model provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0208] The above is a detailed introduction to the training method, device, server and computer-readable storage medium of an image feature extraction model provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A training method for an image feature extraction model, characterized in that: include: Obtaining image features and an image feature extraction model of a training sample, wherein the image feature extraction model includes a convolution module and a global attention module, wherein the convolution module includes a combination filter and a plurality of parallel filters, wherein the parallel filter refers to a filter for performing convolution operations in parallel, and the combination filter refers to a filter obtained by combining a plurality of the parallel filters, and the global attention module is an attention layer for performing global attention calculations or a module including the attention layer; Through each of the parallel filters, a convolution operation is performed on the image features of the training sample to obtain image sub-features corresponding to each of the parallel filters; Inputting all the image sub-features into the global attention module to obtain the image features with global attention, including: inputting all the image sub-features into the global attention module, and for each of the image sub-features, obtaining the image sub-feature with global attention corresponding to the image sub-feature; merging all the image sub-features with global attention to obtain the image features with global attention; According to the image features with global attention, adjusting the convolution module to obtain an adjusted convolution module, including: performing a convolution operation on the image features of the training sample through the combined filter to obtain an overall image feature; according to the gap between the image features with global attention and the overall image feature, adjusting the convolution module to obtain the adjusted convolution module; According to the adjusted convolution module and the preset loss function, a trained image feature extraction model is obtained.
2. The training method of the image feature extraction model according to claim 1, characterized in that: The image feature extraction model further includes an attention module, and before acquiring the image features of the training samples and the image feature extraction model, further includes: Get the initial convolution module; Inputting the image features of the training sample into the attention module to obtain an attention vector; The initial convolution module is adjusted according to the attention vector to obtain the convolution module.
3. The training method of the image feature extraction model according to claim 2, characterized in that: The initial convolution module includes a plurality of initial filters, the attention vector includes a plurality of values corresponding to the plurality of initial filters one by one, and the initial convolution module is adjusted according to the attention vector to obtain the convolution module, comprising: For each of the initial filters, multiply the initial filter by a corresponding value in the attention vector to obtain a parallel filter corresponding to each of the initial filters; According to all the parallel filters, the convolution module is obtained.
4. The training method of the image feature extraction model according to claim 1, characterized in that: The step of inputting all the image sub-features into the global attention module and obtaining, for each image sub-feature, an image sub-feature with global attention corresponding to the image sub-feature comprises: Inputting all of the image sub-features into the global attention module; For each of the image sub-features, determining the image sub-feature as a first image sub-feature, and determining all of the image sub-features except the first image sub-feature as second image sub-features; Determine a first relationship value, a second relationship value, and a characteristic value corresponding to the first image sub-feature, and a first relationship value, a second relationship value, and a characteristic value corresponding to the second image sub-feature, wherein the first relationship value is a Q value, the second relationship value is a K value, and the characteristic value is a V value, and the Q value, K value, and V value are parameter values to be learned determined by an attention mechanism; Performing target operations on the first relationship value corresponding to the first image sub-feature and the second relationship value and feature value corresponding to each of the second image sub-features to obtain multiple operation results; The multiple operation results are added together to obtain the image sub-feature with global attention corresponding to the first image sub-feature.
5. The training method of the image feature extraction model according to claim 1, characterized in that: The step of adjusting the convolution module according to the gap between the image feature with global attention and the total image feature to obtain the adjusted convolution module includes: According to the gap between the image feature with global attention and the total feature of the image, gradient reversal is performed to obtain a gradient reversal result; According to the gradient inversion result, the convolution module is adjusted to obtain the adjusted convolution module.
6. A training device for an image feature extraction model, characterized in that: include: An acquisition unit, used to acquire image features and an image feature extraction model of a training sample, wherein the image feature extraction model includes a convolution module and a global attention module, wherein the convolution module includes a combination filter and a plurality of parallel filters, wherein the parallel filter refers to a filter for performing convolution operations in parallel, and the combination filter refers to a filter obtained by combining a plurality of the parallel filters, and the global attention module is an attention layer for performing global attention calculation or a module including the attention layer; A convolution unit, configured to perform a convolution operation on the image features of the training sample through each of the parallel filters to obtain image sub-features corresponding to each of the parallel filters; The global attention unit is used to input all the image sub-features into the global attention module to obtain the image features with global attention, including: inputting all the image sub-features into the global attention module, and for each of the image sub-features, obtaining the image sub-feature with global attention corresponding to the image sub-feature; merging all the image sub-features with global attention to obtain the image features with global attention; The adjusting unit is used to adjust the convolution module according to the image feature with global attention to obtain an adjusted convolution module, including: performing a convolution operation on the image feature of the training sample through the combined filter to obtain an overall image feature; adjusting the convolution module according to the gap between the image feature with global attention and the overall image feature to obtain the adjusted convolution module; The training unit obtains a trained image feature extraction model according to the adjusted convolution module and the preset loss function.
7. A server, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the training method of the image feature extraction model as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the training method of the image feature extraction model described in any one of claims 1 to 5.
