Data processing method, electronic equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a data processing method, electronic equipment and a medium. In the data processing method, importance scores of all data marks output by a model and similarities among the data marks are determined; when the importance score value of each data mark and the similarity between the data marks meet a pruning condition, performing pruning processing on the data marks output by the model, and when the importance score value of each data mark and the similarity between the data marks meet a pooling condition, performing pooling processing on the data marks output by the model. Therefore, pruning processing or pooling processing is carried out on the data marks output by the model through self-adaptive selection, the processing amount of the data marks can be reduced on the basis of ensuring the identification accuracy of the model, and the balance effect of the identification precision and the calculation amount of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, electronic device and medium. Background Art

[0002] The visual transformer (ViT) obtains image features by performing patch embedding and encoding on the input image, and then classifies the input image based on the image features and outputs the classification results. At present, the ViT model has a high classification accuracy, but there is still much room for improvement in computational complexity and computational efficiency. Among them, the number of image tags (tokens) in the image features affects the computational complexity and computational efficiency of the ViT model. The more image tags (tokens) there are, the greater the computational complexity of the ViT model, and the lower the computational efficiency of the ViT model.

[0003] Existing relevant research shows that the classification result output by the ViT model can be obtained by encoding only some image tokens of the input image, that is, there are some redundant image tokens in all the image tokens of the input image that have little influence on the classification result of the ViT model, resulting in a large compression space for the encoding calculation amount of the input image by the ViT model. Summary of the invention

[0004] In order to solve the problem that the ViT model requires a large amount of calculation for the input image, the embodiments of the present application provide a data processing method, an electronic device and a medium.

[0005] A first aspect of an embodiment of the present application provides a data processing method, the data processing method comprising:

[0006] Acquire data to be processed; acquire first data features output by the first model after inputting the data to be processed into the first model; determine a feature reduction method applicable to the first data feature; corresponding to the feature reduction method applicable to the first data feature being the first reduction method, prune at least part of the first data markers in the first data feature; corresponding to the feature reduction method applicable to the first data feature being the second reduction method, pool at least part of the first data markers in the first data feature.

[0007] Based on the above scheme, by adaptively selecting and pruning or pooling the data markers in the data features, the number of data markers can be reduced while ensuring the accuracy of the model as much as possible, thereby reducing the amount of model calculation while ensuring the accuracy of the model.

[0008] It can be understood that the first model can be a self-attention module in a randomly selected coding block in the first model, for example, the first model can be a self-attention module in the 1st, 4th or 7th coding block in the first model. The first model can be a model obtained by training the model to be processed, such as a trained model obtained by training the visual self-attention model (hereinafter referred to as the model to be processed), hereinafter also referred to as the target model.

[0009] It can be understood that the feature reduction method is a method for reducing the number of data tags. The feature reduction method may include a first reduction method and a second reduction method, wherein the first reduction method may refer to pruning some data tags in the data feature, that is, deleting some first data tags in the first data feature. The second reduction method may refer to pooling some data tags in the data feature, that is, merging some first data features in the first data feature.

[0010] It can be understood that the output of the first model can be the output of any layer in the first model.

[0011] In a possible implementation of the first aspect above, the first model may be obtained by: obtaining sample data and a model to be processed; obtaining a first feature of the output of the model to be processed by inputting the sample data into the model to be processed; and adjusting the parameters of the model to be processed based on the first feature to obtain the first model.

[0012] In some optional instances, the model to be processed is a visual self-attention model, the first model includes a self-attention module and an allocation mechanism module, the sample data is a sample image, and the first feature and reference feature output by the first model include the sample image feature and the sample classification image feature output by the self-attention module.

[0013] In some optional instances, the parameters of the model to be processed are adjusted based on the first feature to obtain the first model, including: adjusting the score value threshold or similarity threshold of the allocation mechanism module based on the sample image features and the sample classification image features to obtain the trained visual self-attention model and the target score value threshold and target similarity threshold of the trained visual self-attention model.

[0014] In an embodiment of the present application, by adding an allocation mechanism module to the randomly selected target coding block, and integrating the pruning processing mechanism and the pooling processing mechanism in the allocation mechanism module, the redundant data tags in the input features can be minimized, so that the accuracy and computational complexity of the balanced model can be achieved based on the trained model. In addition, the allocation mechanism module uses lightweight judgment criteria to determine whether to prune or pool the data tags in the input features. It does not require the introduction of additional learning parameters, can be plug-and-play, can achieve model acceleration, and reduce the cost of further fine-tuning of the model. Among them, model acceleration refers to reducing the computational complexity of the deep neural network model or reducing the running time of the deep neural network model.

[0015] In a possible implementation of the first aspect above, pruning is performed on at least part of the first data markers in the first data feature, including: deleting from the first data feature some data markers whose importance score values ​​are less than a target score value threshold; and pooling is performed on at least part of the first data markers in the first data feature, including: merging from the first data feature some data markers whose similarity is greater than a target similarity threshold.

[0016] In some optional instances, the target score value threshold can be a score value threshold or a similarity threshold of an adjustment allocation mechanism module based on sample image features and sample classification image features to obtain a trained visual self-attention model and a target score value threshold and a target similarity threshold of the trained visual self-attention model.

[0017] In an embodiment of the present application, by pruning or pooling at least part of the first data markers in the first data feature, the number of data markers can be minimized while ensuring the accuracy of the model as much as possible, thereby reducing the computational complexity of the model while ensuring the accuracy of the model.

[0018] In a possible implementation of the first aspect above, the first model is a visual self-attention model, the first model includes a self-attention module, the data to be processed is an image to be processed, and the first data feature output by the first model includes the first image feature output by the self-attention module and a classified image label.

[0019] In some optional instances, the data processing method mentioned in the embodiments of the present application can be used not only for image classification, but also for tasks such as target detection, semantic segmentation, and natural language processing.

[0020] In a possible implementation of the first aspect above, determining a feature reduction method applicable to the first data feature includes: determining a first indicator of the first image feature based on a similarity between the first image feature and the classified image label, wherein the greater the similarity, the greater the value of the first indicator, and determining a second indicator of the first image feature based on an importance score value of the first image feature, wherein the greater the variance of the importance score value, the greater the value of the second indicator; corresponding to a sum of the first indicator and the second indicator being greater than a preset indicator threshold, determining that the feature reduction method applicable to the first image feature is the first reduction method, and corresponding to a sum of the first indicator and the second indicator being less than or equal to the preset indicator threshold, determining that the feature reduction method used by the first image feature is the second reduction method.

[0021] In some optional instances, the first indicator may refer to the distribution variance of the importance score value of the image tag in the first image feature, and the second indicator may refer to the mean of the similarity between the image tag in the first image feature and the classification image tag.

[0022] In a possible implementation of the first aspect, the similarity between the first image feature and the classification image label is determined as follows:

[0023]

[0024] Among them, S cls,i represents the similarity between the image tag i and the classification image tag cls in the first image feature, k cls Represents the key corresponding to the classification image label cls output by the self-attention module, k i represents the key corresponding to the image tag i output by the self-attention module, 2 Indicates l 2 Norm operation.

[0025] In a possible implementation of the first aspect, the importance score of the image tag is determined as follows:

[0026] score i =Attn clsi v i2 ,i∈{1,2,...N}

[0027] Among them, score i Indicates the importance score of image tag i, Attn clsi V represents the component corresponding to image tag i in the attention weight output by the self-attention module for the classified image tag, i Represents the value corresponding to the image tag i output by the self-attention module, 2 Indicates l 2 Norm operation, N represents the number of image tags in the first image feature,

[0028] The attention weight matrix is:

[0029]

[0030] Among them, Attn cls represents the attention weight of the self-attention module for the classification image label output, softmax represents the normalization of the attention weight so that the sum of the attention weight is 1, q cls represents the query of the classified image label output by the self-attention module, K represents the matrix consisting of the keys corresponding to the image labels in the first image feature output by the self-attention module, T represents the matrix transpose operation, and d represents q cls Dimension.

[0031] In a possible implementation of the first aspect above, deleting some data tags whose importance score values ​​are less than a target score value threshold from the first data feature includes: deleting multiple image tags whose importance score values ​​are less than the target score value threshold based on the importance score values ​​of the image tags in the first image feature.

[0032] In a possible implementation of the first aspect, merging some data tags whose similarity is greater than a target similarity threshold from the first data feature includes: merging multiple image tags whose similarity is greater than the target similarity threshold based on the similarity between image tags in the first image feature.

[0033] In a second aspect, an embodiment of the present application provides a data processing device, which includes: a first acquisition module, used to acquire data to be processed; a second acquisition module, used to acquire a first data feature output by the first model after the data to be processed is input into the first model; a determination module, used to determine a feature reduction method applicable to the first data feature; a first processing module, used to prune at least part of the first data markers in the first data feature corresponding to the feature reduction method applicable to the first data feature being the first reduction method; and a second processing module, used to pool at least part of the first data markers in the first data feature corresponding to the feature reduction method applicable to the first data feature being the second reduction method.

[0034] In a possible implementation of the second aspect above, the first processing module is used to delete some data tags whose importance score values ​​are less than the target score value threshold from the first data feature; and the second processing module is used to merge some data tags whose similarity is greater than the target similarity threshold from the first data feature.

[0035] In a possible implementation of the second aspect above, the first model is a visual self-attention model, the first model includes a self-attention module, the data to be processed is an image to be processed, and the first data feature output by the first model includes the first image feature output by the self-attention module and a classified image label.

[0036] In a possible implementation of the second aspect above, a determination module is used to determine a first indicator of the first image feature based on a similarity between the first image feature and the classified image label, wherein the greater the similarity, the greater the value of the first indicator, and to determine a second indicator of the first image feature based on an importance score value of the first image feature, wherein the greater the variance of the importance score value, the greater the value of the second indicator; a first processing module is used to determine that the feature reduction method applicable to the first image feature is the first reduction method when the sum of the first indicator and the second indicator is greater than a preset indicator threshold, and a second processing module is used to determine that the feature reduction method used by the first image feature is the second reduction method when the sum of the first indicator and the second indicator is less than or equal to the preset indicator threshold.

[0037] In a possible implementation of the second aspect, the similarity between the first image feature and the classification image label is determined as follows:

[0038]

[0039] Among them, S cls,i represents the similarity between the image tag i and the classification image tag cls in the first image feature, k cls Represents the key corresponding to the classification image label cls output by the self-attention module, k i represents the key corresponding to the image tag i output by the self-attention module, 2 Indicates l 2 Norm operation.

[0040] In a possible implementation of the second aspect, the importance score of the image tag is determined as follows:

[0041] score i =Attn clsi v i2 ,i∈{1,2,...N}

[0042] Among them, score i Indicates the importance score of image tag i, Attn clsi V represents the component corresponding to image tag i in the attention weight output by the self-attention module for the classified image tag, i Represents the value corresponding to the image tag i output by the self-attention module, 2 Indicates l 2Norm operation, N represents the number of image tags in the first image feature,

[0043] The attention weight matrix is:

[0044]

[0045] Among them, Attn cls represents the attention weight of the self-attention module for the classification image label output, softmax represents the normalization of the attention weight so that the sum of the attention weight is 1, q cls represents the query of the classified image label output by the self-attention module, K represents the matrix consisting of the keys corresponding to the image labels in the first image feature output by the self-attention module, T represents the matrix transpose operation, and d represents q cls Dimension.

[0046] In a possible implementation of the second aspect, the first processing module is configured to delete a plurality of image tags having importance scores less than a target score threshold value according to importance scores of the image tags in the first image feature.

[0047] In a possible implementation of the second aspect, the second processing module is configured to merge, according to the similarity between the image tags in the first image feature, a plurality of image tags having a similarity greater than a target similarity threshold.

[0048] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the data processing method mentioned in the present application.

[0049] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the data processing method mentioned in the present application.

[0050] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising: execution instructions, the execution instructions are stored in a readable storage medium, at least one processor of an electronic device can read the execution instructions from the readable storage medium, and at least one processor executes the execution instructions so that the electronic device implements the data processing method in the first aspect and any possible design of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 According to some examples of the present application, a schematic diagram of an application scenario is shown;

[0052] Figure 2 According to some embodiments, a schematic diagram of a ViT model processing image labeling is shown;

[0053] Figure 3 According to some embodiments, a schematic diagram of a Dynamic ViT model processing redundant image tags is shown;

[0054] Figure 4 According to some embodiments, a schematic diagram of a ViT model processing redundant image tags is shown;

[0055] Figure 5 According to some embodiments, another schematic diagram of a ViT model processing redundant image tags is shown;

[0056] Figure 6 According to some embodiments, a schematic diagram of a framework of a method for acquiring a model in a data processing method is shown;

[0057] Figure 7 A schematic diagram of a process of a model training method is shown;

[0058] Figure 8 A schematic flow chart showing a data processing method;

[0059] Fig. 9 A schematic flow chart of another data processing method is shown;

[0060] Fig.10 A schematic diagram showing the comparison results of the accuracy-computational amount curves of different models for data processing after fine-tuning the target parameters in different models;

[0061] Fig.11 A structural schematic diagram of a data processing device is shown;

[0062] Fig.12 A schematic diagram of the hardware structure of an electronic device is shown. DETAILED DESCRIPTION

[0063] The illustrative embodiments of the present application include, but are not limited to, a data processing method, an electronic device, and a medium.

[0064] It can be understood that the data processing method provided in the embodiments of the present application can be used for deep neural network models used in tasks such as image classification, target detection, semantic segmentation, and natural language processing, wherein the types of deep neural network models are not limited to any deep neural network models such as convolutional neural networks, Transformer (a model based on a multi-head attention mechanism), recursive neural networks, and generative adversarial networks. In some optional instances, the data processing method mentioned in the embodiments of the present application can be used for visual self-attention models in image classification tasks.

[0065] like Figure 1As shown, in a specific implementation, the first electronic device 10 can train the visual self-attention model 11 and then send it to the second electronic device 20, so that the second electronic device 20 can classify the input image based on the trained visual self-attention model 11. The first electronic device 10 and the second electronic device 20 can be any implementable device such as a smart phone, a tablet computer, a smart wearable device or a personal digital assistant.

[0066] For ease of understanding, the technical solution in this application is introduced below by taking the visual transformer (ViT) model (visual self-attention model) to process image data as an example.

[0067] like Figure 2 As shown, the visual self-attention module includes a block mapping module 201 and multiple coding blocks (such as coding block 202, coding block 203, etc.), the block mapping module 201 is serially connected with coding block 202 and coding block 203, and the coding blocks in the multiple coding blocks are serially connected.

[0068] Among them, the block mapping module 201 is used to perform block mapping processing on the input image (vectorization processing on the input image) and output the first image feature. Specifically, the input image can be divided into multiple image regions, and the block mapping module 201 (patch embedding) is used to perform block mapping processing on the multiple image regions to obtain the first image feature (the first image feature includes multiple image tags (tokens)), that is, the input image is represented by the first image feature, each image tag (token) in the first image feature corresponds to each image region, and the first image feature is a general feature of the input image (such as color). For example, the input image R 1 C×H×W (C represents the number of channels of the input image, H represents the height of the input image, and W represents the width of the input image) is divided into N image regions R 2 C×P×P (C represents the number of channels in the image region, and P represents the height and width of the image region). Then, each image region can be projected into a space of dimension D through linear transformation to obtain a one-dimensional vector x corresponding to each image region. p , x p ∈R 2 N×D , that is, an image token in the first image feature. The block mapping module 201 is also used to obtain a classification image token based on the first image feature, wherein the classification image token is an image token used to determine the type of the input image.

[0069] The encoding block may include a self-attention module 2021 and a multi-layer perceptron (MLP) module 2022. The self-attention module 2021 is used to obtain a third image feature based on the first image feature and the classified image tag. The image tag in the third image feature is a combination of the corresponding image tag in the first image feature and all image tags in the first image feature having different degrees of influence on the corresponding image tag.

[0070] While the self-attention module 2021 outputs the third image feature, the attention weight of each image token in the first image feature can be obtained. Among them, the attention weight can represent the correlation between each image token in the first image feature and all image tokens (including itself). In general, the attention weight is determined by self-attention processing of the first image feature based on the parameters (Q, K, V).

[0071] The multi-layer perceptron module 2022 obtains a second image feature based on the first image feature and the third image feature, where the second image feature is a complex feature (such as texture) of the input image.

[0072] Figure 3 According to some embodiments, a schematic diagram of a Dynamic ViT model processing redundant image tags is shown. The redundant image tag processing method processes redundant image tags by image tag pruning. It can be understood that Figure 2 In contrast, the method is implemented based on a DynamicViT model in which a scoring module 304 is provided after each coding block (such as coding block 302) of the ViT model.

[0073] Among them, the scoring module 304 is used to score the importance of the image tags in the second image feature output by the multi-layer perceptron module 3022, that is, to score the influence of each image tag (token) in the second image feature output by the multi-layer perceptron module 3022 on the classification result. The higher the score, the greater the influence (importance) on the classification accuracy of the model. Then, the image tags (tokens) with scores greater than the preset score threshold are retained as the image tags (tokens) in the remaining second image features, and the remaining image tags (tokens) are deleted. It can be understood that since the image tags (tokens) in the second image feature correspond to the image regions mentioned above, the image tags (tokens) with scores lower than the preset score threshold are deleted, that is, the image regions with a smaller influence on the classification accuracy of the input image are deleted, thereby completing the pruning of some image tags.

[0074] Although the above data processing method can ensure the classification accuracy of the model, it does not consider the similarity between image labels and does not reduce redundant image labels among multiple image labels with high similarity. It cannot minimize the number of image labels, that is, it cannot minimize the computational complexity of the model.

[0075] Figure 4 According to some embodiments, a schematic diagram of a ViT model processing redundant image tags is shown. The redundant image tag processing method processes the redundant image tags by image tag pooling. It can be understood that Figure 2 In comparison, this method is based on a model implementation in which a merging module 4023 (ToMe) is inserted between the self-attention module 4021 and the multi-layer perceptron module 4022 in each encoding block (such as encoding block 402) of the ViT model.

[0076] Among them, the merging module 4023 is used to merge the image tags in the first image feature with the attention weight output by the self-attention module 4021, that is, to merge multiple image tags in the first image feature whose similarity is higher than the preset similarity threshold into one merged image tag. The greater the similarity of the two image tags, the smaller the impact of merging into one image tag on the classification accuracy of the model. Then, the merged image tag and the unmerged image tag (including the unmerged image tag in the first image feature output by the block mapping module and the image tag in the third image feature) are input into the multi-layer perceptron module 4022 as the fourth image feature. It can be understood that since the image tags in the fourth image feature correspond to the image regions mentioned above, multiple image tags with similarity higher than the preset similarity threshold are merged, that is, the image regions with less impact on the classification accuracy of the input image are merged, thereby completing the pooling of some image tags.

[0077] In some specific implementations, the image tags in the first image feature may be divided into an image tag set A and an image tag set B. Among them, the similarities between the image tags a1, a2 and the image tag b2 are greater than a preset similarity threshold, the similarity between the image tag a3 and the image tag b4 is greater than the preset similarity threshold, the similarity between the image tag a4 and the image tag b3 is greater than the preset similarity threshold, and the similarity between the image tag a5 and the image tag b5 is greater than the preset similarity threshold.

[0078] Connect image tags a1 and a2 in image tag set A with image tag b2 in image tag set B, connect image tag a3 in image tag set A with image tag b4 in image tag set B, connect image tag a4 in image tag set A with image tag b3 in image tag set B, and connect image tag a5 in image tag set A with image tag b5 in image tag set B.

[0079] If the similarity between image markers a1, a2 and image marker b2 is greater than the similarity between image marker a3 and image marker b4, the similarity between image marker a3 and image marker b4 is greater than the similarity between image marker a4 and image marker b3, and the similarity between image marker a3 and image marker b4 is greater than the similarity between image marker a5 and image marker b5, image markers a1, a2 and image marker b2 can be regarded as an image marker group to be merged, and image marker a3 and image feature b4 can be regarded as an image marker group to be merged.

[0080] The image tags a1, a2, and b2 are combined to obtain a combined image tag ab1, and the image tags a3 and b4 are combined to obtain a combined image tag ab2. The uncombined image tags a4, a5, b3, and b5 are integrated with the combined image tags ab1 and ab2 to obtain a fourth image feature.

[0081] Although the above data processing method can reduce the number of similar image tags, it does not consider the impact of image tags on the accuracy of classification results and does not further remove the redundancy caused by image tags with low importance.

[0082] The methods listed above for processing redundant image tags do not use image tag pruning and image tag pooling strategies efficiently, and do not minimize the number of image tags while ensuring the classification accuracy of the model as much as possible, that is, they do not achieve a better balance between accuracy and computational complexity.

[0083] In order to solve the above problems, some embodiments of the present application provide another method for processing redundant data tags using a model. In this method, for a trained deep neural network model, after the model extracts the data features of the data to be processed, it is determined whether the data features output by the model are pruning or pooling operations. If the difference between the data tags is large, and the average similarity between the data tags and the reference data tags (such as the reference data tags in the image classification scenario can be classification image tags) is high, the data tags in the data features can be pruned. If the difference between the data tags is small, and the average similarity between the data tags and the reference data tags is low, the data tags in the data features can be pooled. In this way, by adaptively selecting to prune or pool the data tags in the data features, the number of data tags can be reduced while ensuring the accuracy of the model as much as possible, thereby reducing the amount of calculation of the model while ensuring the accuracy of the model.

[0084] For example, for the trained ViT model, a distribution mechanism module is set between the self-attention module and the multi-layer perceptron module in some encoding blocks. Figure 5As shown, the allocation mechanism module 5023 can determine whether the image mark corresponding to the current coding block is suitable for pruning operation or pooling operation to reduce the redundancy of the mark. If it is suitable for pruning operation, the allocation mechanism module 5023 can prune the image marks in the first image feature output by the block mapping module 501 and the third image feature output by the self-attention module 5021. If it is suitable for pooling operation, the allocation mechanism module 5023 can pool the image marks in the first image feature output by the block mapping module 501 and the third image feature output by the self-attention module 5021. In this way, by adaptively selecting to prune or pool the image marks based on the image features input to the allocation mechanism module 5023, the number of image marks can be reduced on the basis of ensuring the accuracy of the classification results output by the model, thereby reducing the calculation amount of the model while ensuring the classification accuracy of the model.

[0085] In some optional examples, the determination allocation mechanism module 5023 may determine whether the image mark corresponding to the current coding block is suitable for a pruning operation or a pooling operation to reduce the redundancy of the mark, and the manner may include:

[0086] The allocation mechanism module 5023 can determine the similarity between each image tag in the third image feature and the classification image tag, determine the difference between the image tags in the third image feature by calculating the variance, determine the average similarity between each image tag in the third image feature and the classification image tag by calculating the mean, and then determine the way to reduce tag redundancy.

[0087] For example, the allocation mechanism module 5023 can obtain the query value, key-value pair, importance score value of each image tag in the third image feature output by the self-attention module, as well as the query value and key-value pair of the classification image tag. The similarity between each image tag in the third image feature and the classification image tag can be determined based on the key of each image tag in the third image feature and the key of the classification image tag, and then the average similarity can be determined by calculating the mean to determine the similarity between each image tag and the classification image tag. In addition, the difference between the image tags in the third image feature can be determined based on the importance score value of each image tag in the third image feature by calculating the variance.

[0088] In some optional examples, the allocation mechanism module 5023 performs pruning processing on the image tags in the first image feature output by the block mapping module 501 and the third image feature output by the self-attention module, including:

[0089] The allocation mechanism module 5023 can delete the image tags whose importance scores are lower than the target score threshold in the third image feature based on the importance score values ​​corresponding to each image tag in the third image feature output by the self-attention module 5021, and complete the pruning process. By pruning some image tags (image tags in the third image feature) input to the allocation mechanism module 5023, the number of image tags can be reduced, and the amount of calculation of the model can be reduced.

[0090] In some optional examples, the allocation mechanism module 5023 may perform pooling processing on the first image feature output by the block mapping module 501 and the image tags in the third image feature output by the self-attention module 5021 in the following manner:

[0091] The allocation mechanism module 5023 can combine multiple image tags with similarities higher than the target similarity threshold into one image tag to complete the pooling process for the similarity of the image tags in the third image feature output by the self-attention module 5021. By pooling some image tags (image tags in the third image feature) input to the allocation mechanism module 5023, the number of image tags can be reduced, and the amount of calculation of the model can be reduced.

[0092] In some optional instances, the training method of the ViT model may include:

[0093] The sample image is input into the ViT model to be trained. In the ViT model to be trained, the input image features and sample classification image tags input into each allocation mechanism module 5023 by the self-attention module 5021 are obtained. Based on the input image features and sample classification image tags of each allocation mechanism module 5023, the first index and the second index corresponding to the input image features are determined. Among them, the first index can represent the similarity between each image tag in the input image feature and the sample classification image tag, and the second index can represent the distribution difference degree of the image tag in the input image feature. Based on the first index and the second index corresponding to the input image features of each allocation mechanism module 5023, the parameters (score value threshold, similarity threshold) of each allocation mechanism module 5023 are adjusted until the difference between the classification result obtained by classifying the sample image based on the model with adjusted parameters and the classification result obtained by classifying the sample image based on the model without adjusted parameters is less than the preset difference threshold, and the trained ViT model and the target score value threshold and target similarity threshold of the ViT model are obtained.

[0094] The following is a detailed introduction to the method for obtaining the model (or training method) in the data processing method mentioned in the embodiment of the present application. Figure 6As shown, for the model to be processed, such as the ViT model, a target coding block can be randomly selected, and an allocation mechanism module can be added to the randomly selected target coding block. The allocation mechanism module introduces a lightweight judgment criterion to obtain an efficient ViT model.

[0095] The model training method mentioned in the embodiment of the present application is described in detail below. Figure 7 A flow chart of a model training method is shown, which can be executed by an electronic device or a cloud server. Figure 7 As shown, the model training method may include:

[0096] 701: Obtain a sample data set and a model to be processed, and input the sample data set into the model to be processed.

[0097] It can be understood that the sample data set can be a sample image set, a sample text set, or a sample audio data set. The model to be processed can be a visual self-attention model to be processed. As mentioned above, the model to be processed includes a block mapping module and a plurality of encoding blocks, the block mapping module is serially connected to the encoding block, and the encoding blocks in the plurality of encoding blocks are serially connected. Each encoding block can include a self-attention module, a distribution mechanism module, and a multi-layer perceptron module connected in sequence.

[0098] 702: In each target coding block of the model to be processed, obtain the first data feature and the reference data mark of the allocation mechanism module in each target coding block.

[0099] It can be understood that the target coding block can be a coding block randomly selected from the model to be processed. For example, the target coding block can be the 1st, 4th, and 7th coding blocks in the visual self-attention model. After the sample data set is input into the model to be processed, the output features of the previous module can be used as the input features of the next module connected to it.

[0100] For example, the output features of the block mapping module in the visual self-attention model can be the input features of the first coding block. The output features of the self-attention module in the first coding block can be used as the input features of the allocation mechanism module, and the output features of the block mapping module can also be the input features of the allocation mechanism module. When the first coding block is selected as the target coding block, the input features of the allocation mechanism module (the output features of the self-attention module in the first coding block and the output features of the block mapping module) can be called the first data features.

[0101] It can be understood that the reference data tag can be a data tag obtained by fusing the first data feature, and the reference data tag can include all information of the sample data in the sample data set. For example, in an image classification scenario, the first data feature is an image tag set, and the reference data tag is a classification image tag.

[0102] 703: Determine a first indicator and a second indicator corresponding to the first data feature based on the first data feature of the allocation mechanism module in each target coding block and the reference data mark.

[0103] It can be understood that the first data indicator corresponding to the first data feature can represent the similarity between each data tag in the first data feature and the reference data tag. The second indicator corresponding to the first data feature can represent the difference between data features in the first data feature. In some specific implementations, the difference between data features in the first data feature can be described by the distribution variance of the importance score value of the data tag.

[0104] In some optional instances, the first indicator may be the average similarity between the data markers in the first data feature and the reference data markers. When the average similarity between the data markers in the first data feature and the reference data markers is low, it means that the interaction between the data markers in the first data feature and the reference data markers is not sufficient. In the subsequent processing of the data markers in the first data feature, the data markers in the first data feature should be pooled as much as possible to retain data markers with representative information. When the average similarity between the data markers in the first data feature and the reference data markers is high, it means that the interaction between the data markers in the first data feature and the reference data markers is sufficient. In the subsequent processing of the data markers in the first data feature, they should be pruned as much as possible, that is, only the data markers with high importance in the first data feature should be focused on.

[0105] In some optional examples, the first indicator may be expressed in the form shown in formula (1):

[0106] S OP1 =avg[S tokens ] Formula (1)

[0107] Among them, S op1 can represent the first indicator, tokens can represent the data tag in the first data feature, S tokens It can represent the similarity between a data marker in the first data feature and a reference data marker, and avg[] can represent the operation of finding the average of the similarities between multiple data markers and the reference data marker.

[0108] In the embodiment of the present application, a cosine function can be used to determine the similarity between each data tag in the first data feature and the reference data tag. For example, for the data tag i and the reference data tag cls (such as a classification image tag) in the first data feature, the key corresponding to the data tag i output by the self-attention module and the key corresponding to the reference data tag cls can be obtained, and the similarity between the data tag i and the reference data tag cls can be determined based on the key corresponding to the data tag i and the key corresponding to the reference data tag cls. The specific determination method can be in the form shown in formula (2):

[0109]

[0110] Among them, S cls,i It can represent the similarity between the data label i in the first data feature and the reference data label cls, k cls It can represent the key corresponding to the reference data tag cls output by the self-attention module, k i It can represent the key corresponding to the data tag i output by the self-attention module, 2 Indicates that l 2 Norm operation.

[0111] When the cosine function is used to determine the similarity between each data marker in the first data feature and the reference data marker, the first indicator can be determined in the form shown in formula (3):

[0112]

[0113] Among them, S op1 can represent the first indicator, N can represent the number of data tags in the first data feature, k cls It can represent the key corresponding to the reference data tag cls (such as the classification image tag) output by the self-attention module, k i It can represent the key corresponding to the data tag i output by the self-attention module, 2 Indicates that l 2 Norm operation.

[0114] In some optional examples, the second indicator may be the distribution variance of the importance score values ​​of the data markers in the first data feature. For example, for an image, different data markers have different distributions and different degrees of difficulty in recognition. For an easily recognizable image, the background and the object in the background are easy to distinguish. Therefore, based on the distribution variance of the importance score values ​​of the data markers, only some of the data markers in the first data feature may be focused on, for example, the data markers corresponding to the image region containing the object in the image.

[0115] In the embodiment of the present application, the second index can be expressed in the form shown in formula (4):

[0116] S OP2 =var[scores] formula (4)

[0117] Among them, S op2 can represent the second indicator, scores can represent the importance score value of the data tag in the first data feature, and var[] can represent the variance operation.

[0118] In the embodiment of the present application, the importance score value of the data tag i output by the self-attention module can be obtained, and the importance score value of the data tag i can be expressed in the form shown in formula (5):

[0119]

[0120] Among them, score i It can represent the importance score value of data tag i, Attn clsi It can represent the component corresponding to data tag i in the attention weight output by the self-attention module for the reference data tag, v i It can represent the value corresponding to the data tag i output by the self-attention module. 2 Indicates that l 2 Norm operation, N can represent the number of data tags in the first data feature.

[0121] The attention weight of the self-attention module for the classification image label can be expressed in the form shown in Formula 6:

[0122]

[0123] Among them, Attn cls represents the attention weight output by the self-attention module for the reference data tag (such as the classification image tag), and softmax represents the normalization of the attention weight so that the sum of the attention weights is 1. cls It can represent the query of the reference data label output by the self-attention module, K represents the matrix composed of the keys corresponding to the data labels in the first data feature output by the self-attention module, T represents the matrix transpose operation, and d represents q cls Dimension.

[0124] 704: Based on the first indicator and the second indicator corresponding to the first data feature, adjust the parameters of the allocation mechanism module in each target coding block until a preset adjustment stop condition is met to obtain a target model.

[0125] It can be understood that the preset adjustment stop condition can be that the difference between the processing result obtained by performing task processing on the input data based on the model with adjusted parameters and the processing result obtained by performing task processing on the input data based on the model without adjusted parameters is less than the preset difference threshold. The model without adjusted parameters refers to a trained model without adding an allocation mechanism module, and the processing result obtained by performing task processing on the data to be processed can be used as a standard for adjusting the parameters of the model with added allocation mechanism modules. The preset adjustment stop condition can also be that the number of iterations of the adjustment parameters is greater than the preset iteration number threshold.

[0126] It can be understood that a preset indicator threshold corresponding to the first data feature can be preset, and based on the weighted value of the first indicator and the second indicator corresponding to the first data feature and the size of the preset indicator threshold, a processing mechanism for processing the first data feature by the allocation mechanism module in the target coding block is determined, such as a pruning processing mechanism and a pooling processing mechanism, and the parameters of the allocation mechanism module in the target coding block are adjusted based on the mechanism. Among them, adjusting the parameters of the allocation mechanism module in the target coding block can be adjusting the score threshold or adjusting the similarity threshold to obtain a target score value threshold and a target similarity threshold.

[0127] In some optional instances, if the weighted values ​​of the first indicator and the second indicator corresponding to the first data feature are greater than a preset indicator threshold, some data markers in the first data feature can be deleted based on the current score value threshold to implement pruning of the data markers in the first data feature, and then the model with adjusted parameters is used to perform task processing on the sample data based on the pruned first data feature to obtain a processing result, and the model is retrained based on the processing result until the preset adjustment stop condition is met to obtain the target model.

[0128] If the weighted values ​​of the first indicator and the second indicator corresponding to the first data feature are less than or equal to the preset indicator threshold, some data markers in the first data feature can be merged based on the current similarity threshold to achieve pooling processing of the data markers in the first data feature, and then the model with adjusted parameters is used to perform task processing on the sample data based on the first data feature after pooling processing to obtain a processing result, and the model is retrained based on the processing result until the preset adjustment stop condition is met to obtain the target model.

[0129] In the embodiment of the present application, the weighted values ​​of the first indicator and the second indicator corresponding to the first data feature can be calculated in the manner shown in formula (7):

[0130] S op =λ 1 S op1 +λ 2 S op2 Formula (7)

[0131] Among them, λ 1 It can represent the weight corresponding to the first indicator, λ 2 It can represent the weight corresponding to the second indicator, S op1 Represents the first index, S op2 Represents the second index, S op It can represent the weighted values ​​of the first indicator and the second indicator, and OP can represent the mechanism by which the allocation mechanism module processes the data tags in the first data feature.

[0132] The mechanism by which the allocation mechanism module processes the data tag in the first data feature can be in the form shown in formula (8):

[0133]

[0134] Wherein, Pruning may indicate that the mechanism by which the allocation mechanism module processes the data tags in the first data feature is a pruning processing mechanism, Pooling may indicate that the mechanism by which the allocation mechanism module processes the data tags in the first data feature is a pooling processing mechanism, and S op It can represent the weighted value of the first indicator and the second indicator, and τ can represent the preset indicator threshold corresponding to the first data feature.

[0135] Among them, the pruning processing mechanism is also called data label pruning. In the image classification scenario, data label pruning is based on the differences between image labels, scoring image labels, and deleting image labels with scores below the preset score threshold. This can reduce image labels that have a small impact on the accuracy of the classification results, thereby reducing the amount of computation for the model. The pooling processing mechanism is also called data label pooling. In the image classification scenario, data label pooling is based on the similarity between image labels and reference image labels, aggregating image labels with similarities higher than the preset similarity threshold. This can reduce image labels that have a small impact on the accuracy of the classification results, thereby reducing the amount of computation for the model.

[0136] In the embodiment of the present application, by adding an allocation mechanism module to the randomly selected target coding block and integrating the pruning processing mechanism and the pooling processing mechanism in the allocation mechanism module, it is possible to minimize the redundant data tags in the input features, so that the accuracy and computational complexity of the balanced model can be achieved based on the trained model. In addition, the allocation mechanism module uses lightweight judgment criteria to determine whether to prune or pool the data tags in the input features. It does not require the introduction of additional learning parameters, can be plug-and-play, can achieve model acceleration, and reduce the cost of further fine-tuning of the model. Among them, model acceleration refers to reducing the computational complexity of the deep neural network model or reducing the running time of the deep neural network model.

[0137] The following is a detailed introduction to the process of performing data processing based on the trained model (i.e., the target model) in the data processing method mentioned in the embodiments of the present application. Figure 8 A flow chart of a data processing method is shown, which can be executed by an electronic device or a cloud server. Figure 8 As shown, the data processing method may include:

[0138] 801: Obtain an image to be processed, and input the image to be processed into a target model.

[0139] It can be understood that the image to be processed can be an image in an image database or a frame of an image in a video. Figure 7 The model obtained by the model processing method shown.

[0140] 802: In each target coding block of the target model, obtain a first target image feature and a target classification image label, and input the first target image feature and the target classification image label into a distribution mechanism module in each target coding block.

[0141] It can be understood that when the target model is based on Figure 7 When the model is obtained by the model processing method shown, the target coding block in the target model may be the 1st, 4th, and 7th coding blocks in the target model. The target classification image label may be an image label obtained by fusing the first target image feature, and the target classification image label may include all features of the input image to be processed.

[0142] In some optional examples, the first target image feature of the allocation mechanism module in each target coding block may refer to the output image feature of the previous module of the allocation mechanism module. For example, for the target coding block in the target model, such as the first coding block, the first target image feature of the allocation mechanism module in the target coding block may be the output image feature of the block mapping module and the output image feature of the self-attention module in the target coding block.

[0143] 803: Determine a first indicator and a second indicator corresponding to the first target image feature based on the first target image feature and the target sub-image mark of the allocation mechanism module in each target coding block.

[0144] It can be understood that the method for determining the first indicator and the second indicator corresponding to the first target image feature can adopt the method for determining the first indicator and the second indicator corresponding to the first data feature described above, which will not be repeated here.

[0145] 804: Determine target parameters corresponding to the allocation mechanism modules in each target coding block based on the first indicator and the second indicator corresponding to the first target image feature.

[0146] It can be understood that if the weighted value of the first indicator and the second indicator corresponding to the first target image feature is greater than the preset indicator threshold, Figure 7 The target score value threshold of the allocation mechanism module in the target coding block obtained by the model processing method shown is used as the target parameter corresponding to the allocation mechanism module. If the weighted value of the first indicator and the second indicator corresponding to the first target image feature is greater than or equal to the preset indicator threshold, it can be obtained Figure 7 The target similarity threshold of the allocation mechanism module in the target coding block obtained by the model processing method shown is used as the target parameter corresponding to the allocation mechanism module.

[0147] 805: Based on the target parameters corresponding to the allocation mechanism modules in each target coding block, the first target image feature is processed to obtain a second target image feature.

[0148] In some optional instances, when the target parameter corresponding to the allocation mechanism module in the target coding block is the target score value threshold, the target image mark whose importance score value is lower than the target score value threshold in the first target image feature can be deleted to obtain the second target image feature. For example, for the first coding block, the second target image feature output by the allocation mechanism module in the target coding block may include the remaining target image mark obtained by deleting part of the target image mark from the first target image feature and the image mark in the output image feature of the block mapping module.

[0149] In some optional instances, when the target parameter corresponding to the allocation mechanism module in the target coding block is a target similarity threshold, the target image marks in the first target image feature whose similarity is higher than the similarity threshold can be merged to obtain the second target image feature. For example, for the first coding block, the second target image feature output by the allocation mechanism module in the target coding block may include a merged target image mark obtained by merging some target image marks in the first target image feature, an unmerged target image mark, and an image mark in the output image feature of the block mapping module.

[0150] In the embodiment of the present application, the cosine function can be used to determine the similarity of the target image tag in the first target image feature. For example, for the target image tag i and the target image tag j of the first target image feature, the key (key) in the key-value pair (key-value) of the target image tag i and the key (key) in the key-value pair (key-value) of the target image tag j output by the self-attention module can be obtained, and the similarity between the target image tag i and the target image tag j can be determined based on the key (key) of the target image tag i and the key (key) of the target image tag j. The specific determination method can be in the form shown in formula (9):

[0151]

[0152] Among them, S i,ji It can represent the similarity between the target image tag i and the target image tag j in the first target image feature, k i It can represent the key corresponding to the target image tag i output by the self-attention module, k j It can represent the key corresponding to the target image tag j output by the self-attention module, 2 Indicates that l 2 Norm operation.

[0153] In the embodiment of the present application, by adding the pruning processing mechanism and the pooling processing mechanism integrated by the allocation mechanism module in the randomly selected target coding block, the redundant image tags in the input image features can be minimized, and the classification accuracy and computational complexity of the balanced model can be achieved. In addition, the allocation mechanism module uses lightweight judgment criteria to determine whether to prune or pool the image tags in the input image features. It does not require the introduction of additional learning parameters, can be plug-and-play, can achieve model acceleration, and reduce the cost of further fine-tuning of the model. Among them, model acceleration refers to reducing the computational complexity of the deep neural network model or reducing the running time of the deep neural network model.

[0154] 806: Input the second target image feature into the next module of each target coding block, perform classification processing on the image to be processed, and obtain a classification result.

[0155] It can be understood that the second target image features (output image features) of the allocation mechanism module in each target coding block can be used as the input image features of the multi-layer perceptron module in the corresponding target coding block. The multi-layer perceptron module can perform perception processing on the input image features, that is, the output image features of the comprehensive block mapping module and the second target image features output by the allocation mechanism module, so as to provide image classification processing for subsequent modules to obtain classification results.

[0156] In the embodiment of the present application, by performing pruning or pooling processing on image labels through adaptive selection of image features based on the input allocation mechanism module, the number of image labels can be reduced while ensuring the accuracy of the classification results output by the model, thereby achieving a balanced classification accuracy of the model and reducing the computational complexity of the model.

[0157] The following describes experimental data for verifying the performance of the model processing method mentioned in the embodiments of the present application and the performance of the target model obtained based on the model processing method.

[0158] For example, the model to be processed is a DeiT-S model, the computational effort of the model is 4.6GFLOPs, the image set to be processed is ImageNet, and the accuracy of the classification result output by the DeiT-S model for classifying the images in the image set to be processed is 79.8%. The 3rd, 6th, and 9th coding blocks in the DeiT-S model are replaced with the target coding blocks mentioned in the embodiment of the present application, that is, an allocation mechanism module is added between the self-attention module and the multi-layer perceptron module in the coding block, and the allocation mechanism module uses a lightweight judgment criterion to judge whether to perform pruning or pooling operations on the image tokens in the input image features.

[0159] like Fig. 9 As shown, the input image is input into the DeiT-S model. For the third coding block in the DeiT-S model, when the weighted value of the average similarity between the image token and the classification image token (cls token) in the first image feature of the input allocation mechanism module (Pruning or pooling) and the variance of the importance score value of the image token in the first image feature is greater than the preset indicator threshold, the allocation mechanism module (Pruning or pooling) prunes the image token in the input first image block feature based on the target score value in the trained DeiT-S model to obtain a second image feature.

[0160] When the weighted value of the average similarity between the image token and the classification image block (cls token) in the first image feature input to the allocation mechanism module (Pruning or pooling) and the variance of the importance score value of the image token in the first image feature is less than or equal to the preset indicator threshold, the allocation mechanism module (Pruning or pooling) performs pooling processing on the image token in the input first image feature based on the target similarity threshold in the trained DeiT-S model to obtain a second image feature.

[0161] In an embodiment of the present application, by directly obtaining the target score value or target similarity threshold in the trained DeiT-S model, there is no need to fine-tune it. The number of redundant image tokens in the first image feature can be reduced offline, and the computational complexity of the model can be reduced to 3GFLOPs. At this time, the accuracy of the model is 79.5%.

[0162] In an embodiment of the present application, by obtaining the target score value or target similarity threshold in the trained DeiT-S model and fine-tuning it, for example, by adjusting the target score value (training the trained DeiT-S model again for 30 rounds), the computational complexity of the model can be reduced to 2.7 GFLOPs, and the accuracy of the model is 79.6%.

[0163] It can be understood that the data processing method mentioned in the embodiment of the present application can also be used for other visual attention models such as LV-ViT and T2T-ViT. Table 1 shows the experimental results of the comparison of the computational complexity and accuracy of the model when different label compression models are used to process data in an offline state (without fine-tuning).

[0164] Table 1 Comparative experimental results under off-the-shelf setting

[0165] Model Accuracy Computational Amount (GFLOPs) DeiT-S (benchmark model) 79.8％ 4.6 ATS-DeiT 79.2％ 3 To-Me 79.1％ 3 Trivial PPT 78.4％ 3 PPT 79.5％ 3

[0166] Among them, DeiT-S is a reference model without label compression, ATS-DeiT, To-Me, Trivial PPT, and PPT can represent different label compression methods, PPT can refer to the model obtained by replacing the 3rd, 6th, and 9th coding blocks in the DeiT-S model with the target coding blocks mentioned in the embodiments of the present application, and Trivial PPT refers to the use of pruning and pooling operations in each coding block in the PPT model. It can be seen from Table 1 that the proposed label compression method PPT has achieved the best classification results under the same amount of computation. When pruning and pooling are used simultaneously in each coding block, the classification accuracy of the model output for image classification processing is less than the classification accuracy of the model output for image classification processing obtained by adding an allocation mechanism module (Pruning or pooling) in the coding block.

[0167] Fig.10 The figure shows the curve comparison results of the accuracy-computational amount of data processing by different models after fine-tuning the target parameters in different models. Fig.10As shown in the figure, in the case of fine-tuning the model, compared with other compression labeling algorithms, the proposed method PPT achieves the best classification results with the same amount of computation. In the offline case, PPT is also competitive with existing fine-tuning methods, achieving results comparable to those of the fine-tuned E-ViT. It can be seen that by pruning or pooling image labels (tokens) through adaptive selection of image features based on the input allocation mechanism module, the number of image labels can be reduced relatively more efficiently while ensuring the accuracy of the classification results output by the model as much as possible, achieving a better balance between the classification accuracy of the model and the amount of computation of the model.

[0168] The data processing device mentioned in the embodiment of the present application is described in detail below. Fig.11 A schematic diagram of the structure of a data processing device is shown. Fig.11 As shown, the data processing device may include:

[0169] The first acquisition module 1101 is used to acquire data to be processed;

[0170] The second acquisition module 1102 is used to acquire a first data feature output by the first model after the data to be processed is input into the first model;

[0171] A determination module 1103, configured to determine a feature reduction method applicable to the first data feature;

[0172] A first processing module 1104 is configured to perform pruning processing on at least part of the first data tags in the first data feature, corresponding to the feature reduction method applicable to the first data feature being a first reduction method;

[0173] The second processing module 1105 is configured to perform pooling processing on at least part of the first data tags in the first data feature, corresponding to the feature reduction method applicable to the first data feature being a second reduction method.

[0174] In some optional instances, the first processing module is used to delete some data tags whose importance score values ​​are less than a target score value threshold from the first data feature; and the second processing module is used to merge some data tags whose similarity values ​​are greater than a target similarity threshold from the first data feature.

[0175] In some optional instances, the first model is a visual self-attention model, the first model includes a self-attention module, the data to be processed is an image to be processed, and the first data feature output by the first model includes a first image feature output by the self-attention module and a classified image label.

[0176] In some optional instances, a determination module is used to determine a first indicator of the first image feature based on a similarity between the first image feature and a classified image label, wherein the greater the similarity, the greater the value of the first indicator, and to determine a second indicator of the first image feature based on an importance score value of the first image feature, wherein the greater the variance of the importance score value, the greater the value of the second indicator; a first processing module is used to determine that the feature reduction method applicable to the first image feature is the first reduction method when the sum of the first indicator and the second indicator is greater than a preset indicator threshold, and a second processing module is used to determine that the feature reduction method used by the first image feature is the second reduction method when the sum of the first indicator and the second indicator is less than or equal to the preset indicator threshold.

[0177] In some optional examples, the similarity between the first image feature and the classification image tag is determined as follows:

[0178]

[0179] Among them, S cls,i represents the similarity between the image tag i and the classification image tag cls in the first image feature, k cls Represents the key corresponding to the classification image label cls output by the self-attention module, k i represents the key corresponding to the image tag i output by the self-attention module, 2 Indicates l 2 Norm operation.

[0180] In some optional examples, the importance score of the image tag is determined as follows:

[0181]

[0182] Among them, score i Indicates the importance score of image tag i, Attn clsi V represents the component corresponding to image tag i in the attention weight output by the self-attention module for the classified image tag, i Represents the value corresponding to the image tag i output by the self-attention module, 2 Indicates l 2 Norm operation, N represents the number of image tags in the first image feature,

[0183] The attention weight matrix is:

[0184]

[0185] Among them, Attn cls represents the attention weight of the self-attention module for the classification image label output, softmax represents the normalization of the attention weight so that the sum of the attention weight is 1, qcls represents the query of the classified image label output by the self-attention module, K represents the matrix consisting of the keys corresponding to the image labels in the first image feature output by the self-attention module, T represents the matrix transpose operation, and d represents q cls Dimension.

[0186] In some optional examples, the first processing module is used to delete a plurality of image tags whose importance scores are less than a target score threshold value according to the importance scores of the image tags in the first image feature.

[0187] In a possible implementation of the second aspect, the second processing module is configured to merge, according to the similarity between the image tags in the first image feature, a plurality of image tags having a similarity greater than a target similarity threshold.

[0188] This embodiment is an implementation method corresponding to the above-mentioned method implementation method, and this embodiment can be implemented in conjunction with the method implementation method. The relevant technical details mentioned in the method implementation method are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method implementation method.

[0189] The hardware structure of the electronic device mentioned in this application is introduced below. Fig.12 As shown, a schematic diagram of the hardware structure of an electronic device 100 is shown. It can be understood that the electronic device of the present application can be a server, a desktop computer (desktop computer), a handheld computer, a notebook computer (laptop computer) and other electronic devices. The structure of the electronic device is introduced below by taking the electronic device as an example of a server.

[0190] In one embodiment, the server may include one or more processors 1201, a system control logic 1202 connected to at least one of the processors 1201, a system memory 1203 connected to the system control logic 1202, a non-volatile memory (NVM) 1204 connected to the system control logic 1202, and an input / output (I / O) device 1205 and a network interface 1205 connected to the system control logic 1202.

[0191] In some embodiments, the processor 1201 may include one or more single-core or multi-core processors. In some embodiments, the processor 1201 may include any combination of a general-purpose processor and a dedicated processor (e.g., a graphics processor, an application processor, a baseband processor, etc.). In an embodiment where the server uses an eNB (Evolved Node B, enhanced base station) or a RAN (Radio Access Network, wireless access network) controller, the processor 1201 may be configured to execute various compliant embodiments.

[0192] In some embodiments, system control logic 1202 may include any suitable interface controller to provide any suitable interface to at least one of processors 1201 and / or any suitable device or component in communication with system control logic 1202 .

[0193] In some embodiments, the system control logic 1202 may include one or more memory controllers to provide an interface to the system memory 1203. The system memory 1203 may be used to load and store data and / or instructions 12031. In some embodiments, the memory of the server may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0194] The non-volatile memory (NVM) 1204 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory (NVM) 1204 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (Compact Disc) drive, and a DVD (Digital Versatile Disc) drive.

[0195] The non-volatile memory (NVM) 1204 may include a portion of storage resources on the device where the server is installed, or it may be accessible by the device but not necessarily a portion of the device. For example, the non-volatile memory (NVM) 1204 may be accessed over a network via the network interface 1206 .

[0196] In particular, the system memory 1203 and the non-volatile memory (NVM) 1204 may include: a temporary copy and a permanent copy of the instruction, respectively. The instruction may include: an instruction that causes the server to implement the model processing method or data processing method mentioned in the embodiment of the present application when executed by at least one of the processors 1201. In some embodiments, the instruction, hardware, firmware and / or its software components may be placed in the system control logic 1202, the network interface 1206 and / or the processor 1201 in addition or alternatively.

[0197] The network interface 1206 may include a transceiver for providing a radio interface for the server, and then communicating with any other suitable device (such as a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 1206 can be integrated with other components of the server. For example, the network interface 1206 can be integrated with at least one of the processor 1201, the system memory 1203, the non-volatile memory (NVM) 1204, and a firmware device (not shown) with instructions. When at least one of the processors 1201 executes the instructions, the server implements the model processing method or data processing method mentioned in the embodiments of the present application.

[0198] The network interface 1206 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 1206 may be a network adapter, a wireless network adapter, a telephone modem and / or a wireless modem.

[0199] In one embodiment, at least one of the processors 1201 may be packaged together with logic for one or more controllers of the system control logic 1202 to form a system in package (SiP). In one embodiment, at least one of the processors 1201 may be integrated on the same die with logic for one or more controllers of the system control logic 1202 to form a system on chip (SoC).

[0200] The server may further include: an input / output (I / O) device 1205. The I / O device 1205 may include a user interface to enable a user to interact with the server; the design of the peripheral component interface enables the peripheral component to also interact with the server. In some embodiments, the server also includes a sensor for determining at least one of an environmental condition and location information related to the server.

[0201] In some embodiments, the user interface may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

[0202] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0203] In some embodiments, the sensors may include, but are not limited to, gyroscope sensors, accelerometers, proximity sensors, ambient light sensors, and positioning units. The positioning unit may also be part of or interact with the network interface 1206 to communicate with components of a positioning network (e.g., global positioning system (GPS) satellites).

[0204] The above describes the hardware structure that the electronic device may have. It is understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0205] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0206] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0207] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit, or a microprocessor.

[0208] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0209] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0210] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0211] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0212] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Get the data to be processed; Acquire a first data feature output by the first model after the data to be processed is input into the first model; Determining a feature reduction method applicable to the first data feature; The feature reduction method applicable to the first data feature is a first reduction method, and pruning is performed on at least part of the first data markers in the first data feature; The feature reduction method applicable to the first data feature is the second reduction method, and pooling processing is performed on at least part of the first data tags in the first data feature.

2. The method according to claim 1, characterized in that The pruning of at least part of the first data tags in the first data feature includes: Deleting some data tags whose importance scores are less than a target score threshold from the first data feature; and The performing pooling processing on at least part of the first data tags in the first data features includes: Part of the data tags whose similarity is greater than a target similarity threshold are merged from the first data features.

3. The method according to claim 2, characterized in that The first model is a visual self-attention model, the first model includes a self-attention module, the data to be processed is an image to be processed, and The first data features output by the first model include the first image features and the classification image labels output by the self-attention module.

4. The method according to claim 3, characterized in that The determining of a feature reduction method applicable to the first data feature includes: Determine a first index of the first image feature according to the similarity between the first image feature and the classified image tag, wherein the greater the similarity, the greater the value of the first index, and Determining a second index of the first image feature according to the importance score value of the first image feature, wherein the greater the variance of the importance score value, the greater the value of the second index; Corresponding to the sum of the first indicator and the second indicator being greater than a preset indicator threshold, determining that the feature reduction method applicable to the first image feature is the first reduction method, Corresponding to the sum of the first indicator and the second indicator being less than or equal to the preset indicator threshold, it is determined that the feature reduction method used by the first image feature is the second reduction method.

5. The method according to claim 4, characterized in that The similarity between the first image feature and the classified image mark is determined as follows: Among them, S cls,i represents the similarity between the image tag i and the classification image tag cls in the first image feature, k cls Represents the key corresponding to the classification image label cls output by the self-attention module, k i represents the key corresponding to the image tag i output by the self-attention module, and || ||2 represents the l2 norm operation.

6. The method according to claim 4, characterized in that The importance score of the image tag is determined as follows: Among them, score i Indicates the importance score of image tag i, Attn clsi V represents the component corresponding to image tag i in the attention weight output by the self-attention module for the classified image tag, i represents the value corresponding to the image tag i output by the self-attention module, || ||2 represents the l2 norm operation, and N represents the number of image tags in the first image feature; The attention weight matrix is: Among them, Attn cls represents the attention weight of the self-attention module for the classification image label output, softmax represents the normalization of the attention weight so that the sum of the attention weight is 1, q cls represents the query of the classified image label output by the self-attention module, K represents the matrix consisting of the keys corresponding to the image labels in the first image feature output by the self-attention module, T represents the matrix transpose operation, and d represents q cls Dimension.

7. The method according to claim 2, characterized in that The deleting, from the first data feature, some data tags whose importance scores are less than a target score threshold value, includes: According to the importance score values ​​of the image tags in the first image feature, a plurality of image tags whose importance score values ​​are less than the target score value threshold are deleted.

8. The method according to claim 2, characterized in that: The step of merging the partial data tags whose similarity is greater than a target similarity threshold from the first data features includes: According to the similarity between the image tags in the first image feature, a plurality of image tags whose similarity is greater than the target similarity threshold are merged.

9. A data processing device, characterized in that: The data processing device comprises: A first acquisition module, used to acquire data to be processed; A second acquisition module, used for acquiring a first data feature output by the first model after the data to be processed is input into the first model; A determination module, used to determine a feature reduction method applicable to the first data feature; a first processing module, configured to perform pruning processing on at least part of the first data tags in the first data feature, corresponding to the feature reduction method applicable to the first data feature being a first reduction method; A second processing module is configured to perform pooling processing on at least part of the first data tags in the first data feature, corresponding to the feature reduction method applicable to the first data feature being a second reduction method.

10. The device according to claim 9, characterized in that The first processing module is used to delete some data tags whose importance score values ​​are less than a target score value threshold from the first data feature; and The second processing module is used to merge some data tags whose similarity is greater than a target similarity threshold from the first data features.

11. The device according to claim 10, characterized in that The first model is a visual self-attention model, the first model includes a self-attention module, the data to be processed is an image to be processed, and The first data features output by the first model include the first image features and the classification image labels output by the self-attention module.

12. The device according to claim 11, characterized in that The determination module is used to determine a first indicator of the first image feature according to the similarity between the first image feature and the classified image mark, wherein the greater the similarity, the greater the value of the first indicator, and Determining a second index of the first image feature according to the importance score value of the first image feature, wherein the greater the variance of the importance score value, the greater the value of the second index; The first processing module is configured to determine that a feature reduction method applicable to the first image feature is the first reduction method in response to a sum of the first indicator and the second indicator being greater than a preset indicator threshold; The second processing module is used to determine that the feature reduction method used by the first image feature is the second reduction method, corresponding to the sum of the first indicator and the second indicator being less than or equal to a preset indicator threshold.

13. An electronic device, characterized in that: include: A memory, used to store instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, used to execute the data processing method described in any one of claims 1-8.

14. A readable storage medium, characterized in that: The readable medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the data processing method according to any one of claims 1 to 8.

15. A computer program product, characterized in that include: An execution instruction is stored in a readable storage medium. At least one processor of an electronic device can read the execution instruction from the readable storage medium. The at least one processor executes the execution instruction so that the electronic device implements the data processing method described in any one of claims 1 to 8.

Images