Visual large model Token adaptive optimization method, system and device based on differential evolution and medium

Abstract

The invention discloses a visual large model Token adaptive optimization method, system and device based on differential evolution and a medium, and the method comprises the steps: carrying out the data processing of an image classification data set, an instance segmentation data set and a saliency target detection data set, and obtaining all Tokens corresponding to each image through a Patch Embedding and position coding method; obtaining a plurality of groups of Tokens corresponding to each image through a random selection mode, and performing data processing to output all Tokens corresponding to each image and the plurality of groups of Tokens selected from each image; constructing a Token adaptive selection module, a self-attention optimization module and a downstream task output module; a complete Token adaptive optimization visual large model is constructed; training a reconstruction model and a complete Token self-adaptive optimized visual large model; performing model reasoning to obtain an image classification result, an instance segmentation result image and a saliency target detection result image; the system, the equipment and the medium are used for implementing the method. The method can be widely applied to various visual tasks such as image classification, instance segmentation and saliency target detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a visual large model Token adaptive optimization method, system, device and medium based on differential evolution. BACKGROUND

[0002] Token adaptive optimization in visual large models is an important research direction in the field of computer vision. With the wide application of visual large models in image classification, object detection, semantic segmentation and other tasks, how to efficiently process and optimize Tokens in input images has become a key to improving model performance. Traditional visual models usually use fixed Token representation methods to divide images into equal-sized regions for processing, ignoring the semantic differences and importance of different regions in the image, leading to limitations in model effectiveness. To solve this problem, Token adaptive optimization technology dynamically adjusts the importance of each Token in the task, enabling the model to flexibly select and process Tokens, thereby improving the model's expressive ability. This adaptive mechanism allows the model to more accurately focus on important parts of the image, reducing the interference of redundant information, and can significantly improve the model's performance.

[0003] Existing Token optimization techniques in visual large models significantly improve model performance and computational efficiency through dynamic selection, redundant computation reduction, and multi-scale information fusion strategies, making certain progress. These methods adaptively optimize Token processing methods, reducing unnecessary calculations while maintaining high accuracy, enhancing the model's adaptability in complex tasks. Despite these techniques' progress in improving task performance and robustness, they still face information loss and excessive model complexity. In addition, existing methods usually tightly bind Token selection to specific tasks, lacking sufficient flexibility to meet the needs of diverse tasks.

[0004] Patent application CN118155048A discloses a key feature adaptive selection method based on ViT and reinforcement learning, related to the field of computer vision. This method dynamically adjusts the weight distribution of image Tokens through reinforcement learning, combining the self-attention mechanism of visual large models, enabling the model to automatically select and reinforce key features in images, while calculating global and local feature relationships through multi-head self-attention to improve the model's sensitivity to important information and recognition ability. However, this method directly multiplies Tokens with corresponding weights, which may change the original image information in the patch, affecting the accuracy of feature expression. In addition, this method is only applicable to classification tasks, lacking adaptability to other diverse tasks. At the same time, the interaction between the weight generator and ViT relies on the reinforcement learning process, which undoubtedly increases the complexity and computational burden of model training.

[0005] The patent application with publication number CN118761439A discloses a network structure construction method and device based on knowledge distillation, relating to the technical field of network structure construction. This method optimizes Token selection in visual large models through a Token selection network based on knowledge distillation, and uses teacher network distillation necessity scores to reduce computational burden while maintaining relatively good performance. At the same time, by fusing spatial and channel information, the model efficiency is further improved. However, the disadvantage of this method is that discarding unimportant Tokens leads to a decrease in task accuracy, thereby affecting the overall performance of the model. In addition, this method has only been verified in image classification tasks, lacking adaptability and generalization ability for other task types, limiting its application scope.

[0006] Existing Token adaptive optimization methods are usually only applicable to specific tasks. These methods rely on task-specific rules or conditions, resulting in limited effectiveness when dealing with diverse tasks. In addition, existing methods often discard some Tokens during the Token selection process to reduce computational complexity, which can lead to a decrease in model accuracy, especially in complex scenarios or multi-task environments. SUMMARY

[0007] To overcome the shortcomings of the prior art, the purpose of the present application is to provide a visual large model Token adaptive optimization method, system, device and medium based on differential evolution, which uses a differential evolution algorithm to achieve adaptive optimization. The differential evolution algorithm dynamically selects key information blocks in images through an iterative process, ensuring that each optimization focuses on areas with large amounts of information and avoiding interference from redundant background information. A reconstruction model is used as the objective function for selection and optimization. By introducing the reconstruction model into the objective function of the differential evolution, the selection process of Tokens can be effectively evaluated and guided, ensuring that the selection process is task-independent, making the method more adaptable and widely applicable to different visual tasks. The results of the differential evolution are used to reassign weights to each Token, and these weights are introduced into the self-attention optimization module for model learning. In this way, the model can accurately focus on the information content of Tokens, improving the accuracy of feature learning and effectively improving the performance of subsequent tasks.

[0008] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0009] A visual large model Token adaptive optimization method based on differential evolution, comprising the following steps:

[0010] Step one, data processing is performed on the image classification data set, instance segmentation data set and saliency object detection data set, all Tokens corresponding to each image are obtained through the method of Patch Embedding and position encoding, and multiple groups of Tokens corresponding to each image are obtained through random selection, and all Tokens corresponding to each image and multiple groups of Tokens selected from each image are outputted by data processing;

[0011] Step two, a Token adaptive selection module, a self-attention optimization module and a downstream task output module are constructed; the Token adaptive selection module is realized by a differential evolution algorithm, and the differential evolution algorithm includes a reconstruction model; the self-attention optimization module is a network model based on Transformer; the downstream task output module includes an image classification task output head, an instance segmentation task output head and a saliency object detection task output head;

[0012] Step three, based on the Token adaptive selection module, the self-attention optimization module and the downstream task output module constructed in step two, a complete Token adaptive optimization visual large model is constructed;

[0013] Step four, a reconstruction data set to be trained is obtained, data preprocessing is performed on the reconstruction data set to be trained, a training set of the reconstruction model is obtained, the training set of the reconstruction model is inputted into the reconstruction model in the Token adaptive selection module constructed in step two to reconstruct the original image, the reconstructed original image and the real image are calculated loss, the parameters are updated through back propagation for training, and the corresponding weight file is obtained; a downstream task data set to be trained is obtained, data preprocessing and division are performed on the downstream task data set to be trained, a training set and a test set of the complete Token adaptive optimization visual large model are obtained, the training set of the complete Token adaptive optimization visual large model is inputted into the complete Token adaptive optimization visual large model constructed in step three, the training result of the model is calculated loss, the parameters are updated through back propagation for training, and the trained complete Token adaptive optimization visual large model and the weight file of the image classification task, the weight file of the instance segmentation task and the weight file of the saliency object detection task are obtained, the complete Token adaptive optimization visual large model is constructed, and the results of the image classification task, the instance segmentation task and the saliency object detection task are outputted according to the corresponding weight file;

[0014] Step five, based on the test set of the complete Token adaptive optimization visual large model after step four pretreatment, using the complete Token adaptive optimization visual large model and the weight file of the image classification task, the weight file of the instance segmentation task and the weight file of the saliency object detection task trained in step four, the model inference is carried out through the complete Token adaptive optimization visual large model constructed in step three, and the final image classification result, instance segmentation result image and saliency object detection result image are obtained.

[0015] The specific method of step one comprises:

[0016] Step 1.1, the images in the image classification data set, the instance segmentation data set and the saliency object detection data set are divided into fixed size image blocks through Patch Embedding, and each image block is flattened into a vector and mapped to a high dimensional space through linear transformation to form a serialized embedded representation;

[0017] Step 1.2, add position encoding to the embedded representation of each image block, the dimension of the position encoding is the same as the dimension of the image block embedding, the embedded representation of each image block is added to the corresponding position encoding to obtain the corresponding Tokens; randomly select multiple groups of Tokens from all Tokens, and the number of Tokens selected in each group is one fourth of the total number of Tokens in the image.

[0018] The specific method of step two comprises:

[0019] Step 2.1, the Token adaptive selection module is realized by differential evolution algorithm, and the differential evolution algorithm includes a reconstruction model, which is used as the objective function in the differential evolution algorithm; the Token adaptive selection module is used to find the Tokens coordinates with large amount of information in the image, and the input of the Token adaptive selection module is the multiple groups of Tokens obtained in step one data processing, and the output is the coordinates of the group of Tokens with large amount of information;

[0020] The flow of differential evolution algorithm is:

[0021] The input of the differential evolution algorithm is multiple groups of Tokens, and the output is the coordinates of the group of Tokens with the largest amount of information; each group of Tokens is input into the reconstruction model of the differential evolution algorithm, and multiple reconstruction tensors are generated after processing multiple groups of Tokens, each reconstruction tensor matches the original input tensor in dimension, the loss function between each reconstruction tensor and the original tensor is calculated, and multiple loss values are generated; wherein each group of Tokens is associated with its respective loss value.

[0022] The Tokens coordinates with the minimum loss value are used as the basis for mutation operation on multiple groups of Tokens coordinates, and the coordinates of Tokens from different groups are exchanged for cross operation to obtain new multiple groups of Tokens coordinates, and new multiple groups of Tokens are obtained according to the new multiple groups of Tokens coordinates; the obtained new multiple groups of Tokens are input into the reconstruction model again for image reconstruction, and a new multiple loss values are calculated through the objective function; the new loss value obtained is compared with the loss value of the last round, and the Tokens with the minimum loss value are selected, and the above mutation, cross, loss value calculation through the objective function and selection operations are iterated N times to finally select the Tokens with the minimum loss value, and the coordinates thereof are output to obtain Tokens coordinates with large information amount;

[0023] The reconstruction model is composed of an encoder containing 6 Transformers and a decoder composed of 3 convolutional layers, the encoder is responsible for feature extraction, and the decoder is used for reconstructing the original image; each encoder is composed of a SelfAttention layer, two LayerNorm layers and a Feed Forward layer, the output of the Self Attention layer is connected to the input of the first Layer Norm layer, the output of the first Layer Norm layer is connected to the input of the Feed Forward layer, and the output of the Feed Forward layer is connected to the input of the second LayerNorm layer; the 3 convolutional layers of the decoder are connected in turn;

[0024] In the reconstruction model, each group of Tokens is placed in the corresponding position of the tensor with the same width and height as the original image after being processed by the encoder, and the remaining positions are filled with Gaussian noise, and then the image is restored by the decoder; the reconstruction tensor corresponding to each group of Tokens is output;

[0025] Step 2.2, the self-attention optimization module is a network model based on Transformer, which is used to extract image features, the input of the self-attention optimization module is the Tokens coordinates with large information amount identified by the Token adaptive selection module in step 2.1 and all Tokens processed in step one, and the output of the self-attention optimization module is the image classification dataset feature, the instance segmentation dataset feature and the saliency object detection dataset feature;

[0026] The self-attention optimization module is a network model of a Transformer architecture, which includes multiple encoders, each of which is composed of a SelfAttention layer, two Layer Norm layers, a Feed Forward layer, and an MLP layer. The output of the SelfAttention layer is connected to the input of the first LayerNorm layer, the output of the first LayerNorm layer is connected to the input of the Feed Forward layer, the output of the Feed Forward layer is connected to the input of the second LayerNorm layer, and the output of the second LayerNorm layer is connected to the input of the MLP layer.

[0027] The specific operation of the self-attention optimization module is as follows:

[0028] The Token coordinates with large information quantity identified by the Token adaptive selection module in step 2.1 and all Tokens processed in step one are input into the Self Attention of the self-attention optimization module. The SelfAttention layer includes Evolutionary score and Similarity score. The SelfAttention layer first assigns high scores to Tokens with large information quantity according to the Token coordinates with large information quantity, and assigns low scores to the remaining Tokens with small information quantity, to obtain the Evolutionary score of each Token. In the calculation of self-attention, first perform the query (Q) and key (K) operations, multiply the obtained Similarity score of each Token by the Evolutionary score of each Token as the new score of each Token, then normalize the modified score, thereby obtaining the final attention weight of each Token. The adjusted final attention weight is then applied to the value (V) to complete the self-attention operation, and the Tokens after context perception are generated through the self-attention operation. The Tokens after context perception are input into the first Layer Norm layer for layer normalization to obtain normalized Tokens, the normalized Tokens are input into the Feed Forward layer for nonlinear transformation to obtain nonlinearly transformed Tokens, the nonlinearly transformed Tokens are input into the second Layer Norm layer for secondary normalization to obtain secondary normalized Tokens, and the Tokens after secondary normalization are input into the MLP layer for dimension adjustment of the Tokens. The MLP layer outputs the image classification dataset features, instance segmentation dataset features, and saliency object detection dataset features.

[0029] Step 2.3, the downstream task output module outputs the results of the image classification task, the instance segmentation task and the saliency object detection task according to the image classification dataset features, the instance segmentation dataset features and the saliency object detection dataset features output by the self-attention optimization module in step 2.2 respectively;

[0030] The image classification dataset features, the instance segmentation dataset features and the saliency object detection dataset features output by the self-attention optimization module in step 2.2 are input into the downstream task output module, and the downstream task output module outputs the class of the image, the instance segmentation image and the saliency object detection image according to the image classification task output head, the instance segmentation task output head and the saliency object detection task output head respectively;

[0031] The implementation method of the image classification task is as follows:

[0032] In the image classification task, the dataset is labeled, each image corresponds to a corresponding class label, a single image data is read, the image is adjusted to adapt to the format required by the model, and the three color channels of the image in the dataset are normalized;

[0033] The image classification dataset features output by the self-attention optimization module in step 2.2 are gathered into class Token, and class Token is mapped to a one-dimensional tensor through a layer of multilayer perception. The length of the tensor is the same as the number of categories in the image classification task, and the value of each position in the tensor represents the probability that the image belongs to the corresponding category, completing the final image classification task;

[0034] The implementation method of the instance segmentation task is as follows:

[0035] In the instance segmentation task, the image is adjusted to adapt to the format required by the model, and the three color channels of the image are normalized respectively;

[0036] The instance segmentation dataset features output by the self-attention optimization module in step 2.2 are realized by point-by-point linear mapping or mask transformer through decoding process, and the training adopts pixel-by-pixel cross-entropy loss for end-to-end optimization. The instance segmentation dataset features are up-sampled, and the maximum index operation is applied to determine a unique class for each pixel, and finally the result image of the instance segmentation task is output;

[0037] The implementation method of the saliency object detection task is as follows:

[0038] In the salient object detection task, the image is adjusted to adapt to the format required by the model, and the salient object detection dataset features output by the self-attention optimization module in step 2.2 are sent into the decoder together with the latent variables, the initial value of which comes from the standard normal distribution, and the decoder integrates the salient object detection dataset features and the latent variables to generate a saliency map that identifies the location and range of the salient object in the image.

[0039] The step three specifically includes:

[0040] Step 3.1, input the multiple groups of Tokens obtained in step one into the Token adaptive selection module constructed in step 2.1. The image is divided into small blocks by patch embedding and converted into feature vectors Tokens. Then, a part of Tokens are randomly selected as the initial population of the differential evolution algorithm. The algorithm iterates with the trained reconstruction model as the objective function, which consists of an encoder containing 6 Transformers and a decoder composed of 3 convolutional layers. The encoder is responsible for feature extraction, and the convolutional layers are used for reconstructing the original image. In the reconstruction model, the selected Tokens are processed by the encoder and placed in the corresponding positions of the tensor with the same width and height as the original image according to the coordinates. The remaining positions are filled with Gaussian noise, and then the image is restored by the decoder. The differential evolution algorithm compares the differences between the reconstructed images of different groups of Tokens and the original image by calculating the mean square error to judge the reconstruction effect and achieve adaptive selection of Tokens. Finally, the corresponding coordinates of Tokens with large information quantity in the original image are output.

[0041] Step 3.2, input the Tokens coordinates with large information quantity output by the Token adaptive selection module constructed in step 2.1 and all Tokens obtained by data processing in step one into the self-attention optimization module. The self-attention optimization module uses coordinate recognition to accurately distinguish important and unimportant Tokens. According to the importance evaluation system, each Token is assigned a weight between 0 and 1 as its importance weight value. At the same time, the self-attention learning calculates the correlation scores between Tokens, which are multiplied by the importance weights. The product integrates the Token importance and correlation relationship. Finally, the self-attention optimization module outputs the learned image classification dataset features, instance segmentation dataset features, and salient object detection dataset features through MLP.

[0042] Step 3.3, the image classification dataset features, instance segmentation dataset features and saliency object detection dataset features output by the self-attention optimization module constructed in step 2.2 are transmitted to a downstream task output module, and the downstream tasks include image classification, instance segmentation and saliency object detection; in the image classification task, the probabilities of each category of the picture are calculated through a multilayer perceptron, and finally the category with the maximum probability is output; in the instance segmentation task, based on an end-to-end optimization strategy, a unique classification is accurately determined for each pixel in the image, and an instance segmentation map is generated; and in the saliency object detection task, the saliency object detection dataset features output by the self-attention optimization module in step 2.2 are input into the decoder together with the latent variable to generate a saliency map.

[0043] The specific method of the step four preprocessing includes:

[0044] The reconstructed dataset to be trained is read, the reconstructed dataset image is adjusted to 3x224x224 and converted into a format conforming to the reconstructed model, and the three color channels of the reconstructed dataset image are normalized respectively;

[0045] The image classification dataset to be trained is read, the image classification dataset image is adjusted to 3x224x224 and converted into a format conforming to the complete Token adaptive optimization image classification visual large model, and the three color channels of the image classification dataset image are normalized respectively;

[0046] The instance segmentation dataset to be trained is read, the instance segmentation dataset image is adjusted to 3x480x480 and converted into a format conforming to the complete Token adaptive optimization instance segmentation visual large model, and the three color channels of the instance segmentation dataset image are normalized respectively;

[0047] The saliency object detection dataset to be trained is read, the saliency object detection dataset image is adjusted to 3x384x384 and converted into a format conforming to the complete Token adaptive optimization saliency object detection visual large model, and the three color channels of the saliency object detection dataset image are normalized respectively;

[0048] The loss function is used to evaluate the performance of the network model, and the loss functions defined for the reconstruction model and various task models are as follows:

[0049] For the training of the reconstruction model, the following mean square error loss function is adopted:

[0050]

[0051] Wherein, N is the number of samples, y i is the true value of the i-th sample, is the predicted value of the i-th sample, and the mean square error loss function is used to calculate the square of the difference between the predicted value and the true value of each sample, and then the average of the square errors of all samples is calculated;

[0052] For the training of the complete Token adaptive optimization image classification visual large model, the following cross-entropy loss function is used:

[0053] L = -log(p y )

[0054]

[0055] where P y is the probability that the model predicts the sample to belong to the true class y, and the relationship between the probability of predicting the correct class and the loss is quantified by taking the negative logarithm, the lower the probability, the greater the loss; in the formula P y , z y is the output score of the model for the true class y, and z j is the output score for all C classes, and this formula is essentially a softmax function, which converts the original output score of the complete Token adaptive optimization image classification visual large model into a probability distribution, so that the sum of the probabilities of all classes is 1;

[0056] For the instance segmentation task, the following pixel-wise cross-entropy loss function is used:

[0057]

[0058] where N is the number of pixels in the image, each pixel has C classes, and the model is used to predict the probability distribution q i =(q i1 , q 12 , …, q iC ) of each pixel belonging to each class, where i = 1, 2, 3, …, N, and the true label p i is a one-hot vector representing the true class of the pixel, and the total loss of an image is the average of all pixel losses;

[0059] For the salient object detection task, the following binary cross-entropy loss function is used:

[0060]

[0061] where N is the number of pixels in the image, y is the true label of the sample, and takes the value of 0 or 1; when y = 1, it means that the sample belongs to the positive class; when y = 0, it means that the sample belongs to the negative class; is the probability that the model predicts the sample to belong to the positive class, and the value range is [0, 1].

[0062] The step five specifically comprises:

[0063] Step 5.1, load the complete Token adaptive optimization visual large model structure, read the model weight file corresponding to the structure after complete training iteration, and load it into the complete Token adaptive optimization visual large model structure, set the complete Token adaptive optimization visual large model as an inference model, and fix the parameters of each network structure;

[0064] Step 5.2, the test set of the complete Token adaptive optimization visual large model includes an image classification test set, an instance segmentation test set and a saliency target detection test set, the image classification test set, the instance segmentation test set and the saliency target detection test set are respectively input into the corresponding complete Token adaptive optimization visual large model, the complete Token adaptive optimization visual large model firstly realizes adaptive selection of key information blocks in the image by using a Token adaptive selection module to obtain Token position coordinates with large information amount; secondly, the self-attention optimization module is used to adaptively adjust the attention weight according to the information amount of each image block, so as to ensure that more computing resources and attention are concentrated in the region with large information amount in the training process, and the region with small information amount is not concerned, and the features are accurately extracted; finally, the image classification dataset features, the instance segmentation dataset features and the saliency target detection dataset features extracted are input into the corresponding output head according to the specific task in the downstream task output module, and the corresponding output results, i.e. image classification result map, instance segmentation result map and saliency target detection result map, are obtained.

[0065] Step 5.3, compare the complete Token adaptive optimization image classification visual large model with the image classification visual large model, and judge the model results by Top1 accuracy, the larger the value, the more accurate the model classification result.

[0066] The application also provides a visual large model Token adaptive optimization system based on differential evolution, comprising:

[0067] The data processing module is used for data processing of the image classification dataset, the instance segmentation dataset and the saliency target detection dataset, and all Tokens corresponding to each image are obtained by the method of Patch Embedding and position coding; a plurality of groups of Tokens corresponding to each image are obtained by a random selection method, and all Tokens corresponding to each image and a plurality of groups of Tokens selected from each image are output by the data processing.

[0068] The first module is used for constructing a Token adaptive selection module, a self-attention optimization module and a downstream task output module; the Token adaptive selection module is realized by a differential evolution algorithm, and the differential evolution algorithm comprises a reconstruction model; the self-attention optimization module is a network model based on a Transformer; and the downstream task output module comprises an image classification task output head, an instance segmentation task output head and a saliency object detection task output head.

[0069] The model construction module is used for realizing the Token adaptive optimization visual large model by using the Token adaptive selection module, the self-attention optimization module and the downstream task output module.

[0070] The model training module is used for obtaining a reconstruction data set to be trained, performing data preprocessing on the reconstruction data set to be trained, obtaining a training set of the reconstruction model, inputting the training set of the reconstruction model into the reconstruction model in the Token adaptive selection module to reconstruct an original image, calculating a loss between the reconstructed original image and a real image, updating parameters by back propagation for training, obtaining a corresponding weight file, obtaining a downstream task data set to be trained, performing data preprocessing and division on the downstream task data set to be trained, obtaining a training set and a test set of the complete Token adaptive optimization visual large model, inputting the training set of the complete Token adaptive optimization visual large model into the complete Token adaptive optimization visual large model, calculating a loss between a training result of the model and real data, updating parameters by back propagation for training, obtaining the trained complete Token adaptive optimization visual large model and weight files of the image classification task, the instance segmentation task and the saliency object detection task, constructing the complete Token adaptive optimization visual large model and outputting results of the image classification task, the instance segmentation task and the saliency object detection task according to the corresponding weight files.

[0071] The model inference module is used for realizing the test set of the complete Token adaptive optimization visual large model after preprocessing, utilizing the trained complete Token adaptive optimization visual large model and the weight files of the image classification task, the instance segmentation task and the saliency object detection task, performing model inference by using the complete Token adaptive optimization visual large model, and obtaining final image classification results, instance segmentation result images and saliency object detection result images.

[0072] The application further provides a Token adaptive optimization device of a visual large model based on differential evolution, comprising:

[0073] Memory: store the computer program of the above-mentioned one kind based on differential evolution visual big model Token adaptive optimization method, for computer readable equipment;

[0074] Processor: for executing the computer program realizes the one kind based on differential evolution visual big model Token adaptive optimization method.

[0075] The application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program can realize the one kind based on differential evolution visual big model Token adaptive optimization method when the processor executes.

[0076] Compared with the prior art, the beneficial effects of the present application are:

[0077] 1、The application constructs the reconstruction model in the Token adaptive selection module through step 2.1, and reconstructs the tensor through the multiple groups of Tokens processed by step one, compares the reconstructed tensor with the original tensor and calculates the loss as the basis for selection. This reconstruction step is only for the data set, and is not limited to the task, so using the reconstruction model as the objective function for selection and optimization can effectively evaluate and guide the selection process of the Token, ensure that the selection process is independent of the task, and make the method have stronger adaptability and be widely applicable to different visual tasks.

[0078] 2、The Token adaptive selection module constructed by step 2.1 realizes the adaptive selection of important Tokens in the image through the iterative process of the differential evolution algorithm. The evolution process iterates multiple times on the multiple groups of Tokens obtained by step one data processing, each iteration including mutation, crossover, loss calculation and selection, so that the model can focus on the area with the most information in the image and reduce the interference of redundant background information on feature extraction.

[0079] 3、The application constructs the self-attention optimization module of step 2.2, takes all Tokens in step one data processing and Tokens with large information quantity obtained by the Token adaptive selection module of step 2.1 as input, and accurately distinguishes Tokens with large and small information quantity. This process allows the model to prioritize important image features by allocating more attention to Tokens with large information quantity, while reducing the influence of Tokens with small information quantity. Therefore, this module can focus attention on the area with large information quantity in the input, thereby improving the overall performance of the visual task.

[0080] In summary, the present application constructs a complete Token adaptive optimization visual large model based on the Token adaptive selection module, the self-attention optimization module and the downstream task output module constructed in step two, and significantly improves the performance of the model in multiple visual tasks by introducing reconstruction model, differential evolution algorithm and self-attention optimization technology. The present application reduces the interference of redundant information and optimizes the feature extraction process; at the same time of improving the feature learning accuracy, it makes up for the shortcomings of the existing method in Token adaptive selection and feature extraction; and can be widely applied to image classification, instance segmentation, saliency object detection and other visual tasks, and has strong flexibility and adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1 is the reconstruction model training flowchart of the present application.

[0082] Figure 2 is the complete Token adaptive optimization visual large model training flowchart of the present application.

[0083] Figure 3 is the complete inference flowchart of the present application.

[0084] Figure 4 is the Token adaptive selection module structure diagram of the present application.

[0085] Figure 5 is the reconstruction model structure diagram of the present application.

[0086] Figure 6 is the self-attention optimization module structure diagram. DETAILED DESCRIPTION

[0087] In order to make the implementation process and features of the method clearer and easier to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0088] A visual large model Token adaptive optimization method based on differential evolution includes the following steps:

[0089] Step one, data processing is performed on image classification dataset, instance segmentation dataset and saliency object detection dataset, and all Tokens corresponding to each image are obtained through the method of Patch Embedding and position encoding; through random selection, multiple groups of Tokens corresponding to each image are obtained, and all Tokens corresponding to each image and multiple groups of Tokens selected from each image are outputted by data processing;

[0090] The specific method of step one includes:

[0091] Step 1.1, the images in the image classification dataset, instance segmentation dataset and saliency object detection dataset are segmented into fixed size image blocks by Patch Embedding, and each image block is flattened into a vector and mapped to a high dimensional space by linear transformation to form a serialized embedding representation; the image is converted into sequence data suitable for visual large model.

[0092] (2) Add position encoding to the embedding representation of each image block, which is usually a learnable parameter. The dimension of the position encoding is the same as the dimension of the image block embedding. The embedding representation of each image block is added to the corresponding position encoding to obtain the corresponding Tokens; randomly select multiple groups of Tokens from all Tokens, and the number of selected Tokens in each group is one quarter of the total number of Tokens in the image, thereby preserving the spatial structure information of the image. Provide complete context information for subsequent model processing.

[0093] Step two, build Token adaptive selection module, self-attention optimization module and downstream task output module; the Token adaptive selection module is realized by differential evolution algorithm, and the differential evolution algorithm includes a reconstruction model; the self-attention optimization module is a network model based on Transformer; the downstream task output module includes image classification task output head, instance segmentation task output head and saliency object detection task output head;

[0094] The specific method of step two includes:

[0095] Step 2.1, the Token adaptive selection module is realized by differential evolution algorithm, and the differential evolution algorithm includes a reconstruction model, which is used as the objective function in the differential evolution algorithm; the Token adaptive selection module is used to find the coordinates of Tokens with large information quantity in the image, and the structure of the Token adaptive selection module is as shown in Figure 4 The input of the Token adaptive selection module is the multiple groups of Tokens obtained by step one data processing, and the output is the coordinates of the group of Tokens with large information quantity.

[0096] The flow of differential evolution algorithm is as follows:

[0097] The input of the differential evolution algorithm is multiple groups of Tokens, and the output is the coordinates of the group of Tokens with the largest information quantity; each group of Tokens is input into the reconstruction model of the differential evolution algorithm, and multiple reconstruction tensors are generated after processing multiple groups of Tokens. Each reconstruction tensor matches the original input tensor in dimension, and the loss function between each reconstruction tensor and the original tensor is calculated to generate multiple loss values; these loss values form the basis of the evolution process, and each group of Tokens is associated with its respective loss value;

[0098] On the basis of the group of Tokens coordinates with the minimum loss value, the mutation operation is performed on the multiple groups of Tokens coordinates, the crossover operation is performed by exchanging the coordinates of Tokens from different groups, new multiple groups of Tokens coordinates are obtained, and new multiple groups of Tokens are obtained according to the new multiple groups of Tokens coordinates; the obtained new multiple groups of Tokens are input into the reconstruction model again for image reconstruction, and a new multiple loss values are calculated through the objective function; the new loss value obtained is compared with the loss value of the last round, and the group of Tokens with the minimum loss value is selected, and through the cyclic iteration operation of the mutation, the crossover, the loss calculation and the comparison selection, the purpose of evolution is realized, finally the group of Tokens with the minimum loss value is selected, and the coordinates thereof are output, and the Tokens coordinates with large information amount are obtained;

[0099] The adaptive selection of the key information blocks in the image is realized through the iteration process of the differential evolution algorithm, and the interference of redundant background information on feature extraction is reduced. The strategy simulates the evolution process, optimizes the selection standard of Tokens, and enables the model to focus on the area with the most information amount in the image.

[0100] The differential evolution algorithm can realize adaptive optimization of Tokens in a multi-task environment, significantly improve the precision of the model, and effectively avoid the precision loss caused by discarding Tokens in the traditional method.

[0101] As shown in Figure 5 The reconstruction model is composed of an encoder containing 6 Transformers and a decoder composed of 3 convolutional layers, the encoder is responsible for feature extraction, and the decoder is used for reconstructing the original image; each encoder is composed of a SelfAttention layer, two Layer Norm layers and a Feed Forward layer, the output of the SelfAttention layer is connected to the input of the first LayerNorm layer, the output of the first Layer Norm layer is connected to the input of the Feed Forward layer, and the output of the Feed Forward layer is connected to the input of the second LayerNorm layer; the 3 convolutional layers of the decoder are connected in turn;

[0102] In the reconstruction model, each group of Tokens is placed in the corresponding position of the tensor with the same width and height as the original image after being processed by the encoder, and the remaining positions are filled with Gaussian noise, and then the image is restored by the decoder; the reconstruction tensor corresponding to each group of Tokens is output.

[0103] The reconstruction model is used as the objective function of the differential evolution algorithm for evaluating the quality of the Token population. Since the reconstruction model is selected as the evaluation standard, the Token adaptive selection process is task-independent and can adapt to the needs of different tasks.

[0104] Step 2.2, the self-attention optimization module is a network model based on Transformer, which is used to extract image features. The input of the self-attention optimization module is the Tokens coordinates with large amount of information identified by the Token adaptive selection module in step 1.1 and all Tokens after data processing in step 1. The output of the self-attention optimization module is the image classification dataset features, instance segmentation dataset features and saliency object detection dataset features;

[0105] As shown in Figure 6 , the self-attention optimization module is a network model with Transformer architecture, which contains multiple encoders. Each encoder is composed of a SelfAttention layer, two Layer Norm layers, a FeedForward layer and an MLP layer. The output of the SelfAttention layer is connected to the input of the first Layer Norm layer. The output of the first Layer Norm layer is connected to the input of the FeedForward layer. The output of the FeedForward layer is connected to the input of the second Layer Norm layer. The output of the second Layer Norm layer is connected to the input of the MLP layer.

[0106] The specific operation of the self-attention optimization module is as follows:

[0107] The Tokens with large information quantity identified by the Token adaptive selection module in step 2.1 and all Tokens processed in step one are input into the Self Attention of the self-attention optimization module. The Self Attention layer includes Evolutionary score and Similarity score. The Self Attention layer first assigns high scores to Tokens with large information quantity according to the Tokens coordinates with large information quantity, and assigns low scores to the remaining Tokens with small information quantity, to obtain the Evolutionary score of each Token. In the calculation of self-attention, first perform the query (Q) and key (K) operations, multiply the obtained Similarity score of each Token by the Evolutionary score of each Token as the new score of each Token, and then normalize the modified score to obtain the final attention weight of each Token. The adjusted final attention weight is then applied to the value (V) to complete the self-attention operation, and the Tokens after context perception are generated through the self-attention operation. The Tokens after context perception are input into the first Layer Norm layer for layer normalization to obtain normalized Tokens, and the normalized Tokens are input into the Feed Forward layer for nonlinear transformation to obtain nonlinearly transformed Tokens. The nonlinearly transformed Tokens are input into the second Layer Norm layer for secondary normalization to obtain secondary normalized Tokens. The Tokens after secondary normalization are input into the MLP layer for dimension adjustment of Tokens, and the MLP layer outputs image classification dataset features, instance segmentation dataset features, and saliency object detection dataset features.

[0108] This process allows the model to allocate more attention to Tokens with large information quantity, giving priority to features with large information quantity on the dataset image, while reducing the influence of Tokens with small information quantity. Therefore, the self-attention optimization module can focus its attention on the input area with large information quantity, thereby improving its overall performance.

[0109] Each encoder adaptively adjusts the attention weight according to the information quantity of each image block during learning, ensuring that more computational resources and attention are focused on areas with large information quantity during the training process, rather than on areas with small information quantity. This attention mechanism does not depend on model-specific requirements, but adjusts attention through the feature information quantity of each image block, thereby improving task accuracy.

[0110] Step 2.3, the downstream task output module outputs the results of the image classification task, the instance segmentation task and the saliency object detection task according to the image classification dataset features, the instance segmentation dataset features and the saliency object detection dataset features output by the self-attention optimization module in step 2.2.

[0111] The image classification dataset features, the instance segmentation dataset features and the saliency object detection dataset features output by the self-attention optimization module in step 2.2 are input into the downstream task output module, and the downstream task output module outputs the class of the image, the instance segmentation image and the saliency object detection image according to the image classification task output head, the instance segmentation task output head and the saliency object detection task output head respectively;

[0112] The implementation method of the image classification task is:

[0113] Parameter setting: in the image classification task, CIFAR10, CIFAR100 and Flowers02 dataset are labeled, each image corresponds to a corresponding class label, torchvision library is used to read single image data, the image is adjusted to 3x224x224 and converted to tensor format to meet the input requirements of the model; the mean of 0.485, 0.456, 0.406 and the variance of 0.229, 0.224, 0.225 of the three color channels of the image are used for normalization processing;

[0114] The image classification dataset features output by the self-attention optimization module in step 2.2 are gathered into class Token, class Token is mapped to a one-dimensional tensor through a multilayer perception, the length of the tensor is the same as the number of categories in the image classification task, and the value of each position in the tensor represents the probability that the image belongs to the corresponding category, thereby completing the final image classification task.

[0115] The implementation method of the instance segmentation task is:

[0116] Parameter setting: in the instance segmentation task, the image is adjusted to 3x480x480 and converted to tensor format, the mean of 123.675, 116.28, 103.53 and the variance of 58.395, 57.12, 57.375 are used for normalization processing of the three color channels of the image respectively;

[0117] The instance segmentation dataset feature output by the self-attention optimization module in step 2.2 is linearly mapped or mask transformed point by point through the decoding process, and the training is performed using the pixel-wise cross-entropy loss for end-to-end optimization, the instance segmentation dataset feature is up-sampled, and the maximum index operation is applied to determine the unique class for each pixel, and finally the result image of the instance segmentation task is output;

[0118] The implementation method of the salient object detection task is:

[0119] Parameter setting: in the salient object detection task, the image is adjusted to 3*384*384, the image is converted into a tensor and normalized (mean: [0.485, 0.456, 0.406], standard deviation: [0.229, 0.224, 0.225]);

[0120] The salient object detection dataset feature output by the self-attention optimization module in step 2.2 is sent into the decoder together with the latent variable, and the initial value of the latent variable comes from the standard normal distribution, and the decoder integrates the salient object detection dataset feature and the latent variable to generate a saliency map, which identifies the location and range of the salient object in the image.

[0121] Step three, based on the token adaptive selection module, the self-attention optimization module and the downstream task output module constructed in step two, a complete token adaptive optimization visual large model is constructed;

[0122] Step 3.1, input the multiple groups of Tokens obtained in step one into the Token adaptive selection module constructed in step 2.1, the image is divided into small blocks and converted into feature vector Token through patch embedding, then a part of Token is randomly selected as the initial population of the differential evolution algorithm, and the algorithm iterates with the trained reconstruction model as the objective function, the reconstruction model is composed of an encoder containing 6 Transformers and a decoder composed of 3 convolutional layers, the encoder is responsible for feature extraction, and the convolutional layer is used for reconstructing the original image; in the reconstruction model, the selected Token is placed in the corresponding position of the tensor with the same width and height as the original image after being processed by the encoder, and the remaining positions are filled with Gaussian noise, and then the image is recovered by the decoder; the differential evolution algorithm compares the differences between the reconstructed images of different groups of Token and the original image by calculating the mean square error to judge the reconstruction effect, realizes Token adaptive selection, and finally outputs the corresponding coordinates of the Tokens with large information quantity in the original image;

[0123] Step 3.2, the information amount of the Token coordinates output by the Token adaptive selection module constructed in step 2.1 and all Tokens obtained by data processing in step one are input into the self-attention optimization module. The self-attention optimization module uses coordinate recognition to accurately distinguish important and unimportant Tokens. According to the importance evaluation system, each Token is assigned a weight between 0 and 1 as its important weight value to reflect its importance in image feature expression. At the same time, the self-attention learning calculates the correlation scores between Tokens, which are multiplied by the important weights. The product integrates Token importance and correlation. Finally, the self-attention optimization module outputs the learned image classification dataset features, instance segmentation dataset features, and saliency object detection dataset features through MLP. The self-attention optimization module greatly improves the efficiency and accuracy of image feature extraction, providing high-quality feature expression for subsequent image processing tasks.

[0124] Step 3.3, the image classification dataset features, instance segmentation dataset features, and saliency object detection dataset features output by the self-attention optimization module constructed in step 2.2 are transmitted to the downstream task output module for result output. The downstream tasks include image classification, instance segmentation, and saliency object detection. In the image classification task, the probabilities of each category of the image are calculated through a multi-layer perceptron, and the category with the highest probability is finally output. In the instance segmentation task, based on the end-to-end optimization strategy, each pixel in the image is accurately classified to generate an instance segmentation map. In the saliency object detection task, the saliency object detection dataset features output by the self-attention optimization module in step 2.2 are input into the decoder together with the latent variables to generate a saliency map.

[0125] Step four, obtain the reconstruction dataset to be trained, and perform data preprocessing on the reconstruction dataset to be trained to obtain the training set of the reconstruction model, as shown in Figure 1 The training set of the reconstruction model is input into the reconstruction model in the Token adaptive selection module constructed in step two to reconstruct the original image. The loss between the reconstructed original image and the real image is calculated, and the parameters are updated through backpropagation for training to obtain the corresponding weight file. Obtain the downstream task dataset to be trained, perform data preprocessing and division on the downstream task dataset to be trained, and obtain the training set and test set of the complete Token adaptive optimization visual large model, as shown in Figure 2As shown, the training set of the complete Token adaptive optimized visual large model is input into the complete Token adaptive optimized visual large model constructed in step three, the prediction result of the model is calculated with the real data to obtain the loss, the parameters are updated by back propagation for training, and the trained complete Token adaptive optimized visual large model and the weight file of the image classification task, the weight file of the instance segmentation task and the weight file of the saliency object detection task are obtained, the complete Token adaptive optimized visual large model is constructed, and the results of the image classification task, the instance segmentation task and the saliency object detection task are output according to the corresponding weight file.

[0126] The specific method of the step four preprocessing includes:

[0127] The reconstruction data set to be trained is read through the torchvision library, the reconstruction data set image is adjusted to 3x224x224 and converted into a tensor format conforming to the reconstruction model, and the mean of 0.485, 0.456 and 0.406 and the variance of 0.229, 0.224 and 0.225 are used to normalize the three color channels of the reconstruction data set image respectively;

[0128] The image classification data set to be trained is read through the torchvision library, the image classification data set image is adjusted to 3x224x224 and converted into a tensor format conforming to the complete Token adaptive optimized image classification visual large model, and the mean of 0.485, 0.456 and 0.406 and the variance of 0.229, 0.224 and 0.225 are used to normalize the three color channels of the image classification data set image respectively;

[0129] The instance segmentation data set to be trained is read through the torchvision library, the instance segmentation data set image is adjusted to 3x480x480 and converted into a tensor format conforming to the complete Token adaptive optimized instance segmentation visual large model, and the mean of 123.675, 116.28 and 103.53 and the variance of 58.395, 57.12 and 57.375 are used to normalize the three color channels of the instance segmentation data set image respectively;

[0130] The significant target detection dataset to be trained is read through the torchvision library, the significant target detection dataset image is adjusted to 3*384*384, and converted into a tensor format conforming to the complete Token adaptive optimization significant target detection visual large model, the three color channels of the significant target detection dataset image are normalized using the mean of 0.485, 0.456 and 0.406 and the variance of 0.229, 0.224 and 0.225 respectively;

[0131] In the corresponding network structure of the application, the reconstruction model in the Token adaptive selection module needs to be trained separately, and the self-attention optimization module and the downstream task output module are trained together (different tasks correspond to different models). The reconstruction model uses AdamW as the parameter optimizer, and the learning rate is set to 0.001. The training of the complete Token adaptive optimization large model is different because of different tasks, and the training parameters are different, as follows: in the image classification task, the model uses AdamW as the parameter optimizer, the initial learning rate is set to 0.0001, the epoch is 100, and the single training batch size is 8; in the instance segmentation task, the model uses the stochastic gradient descent method as the optimizer, the initial learning rate is set to 0.001, and the "polynomial" learning rate decay strategy is used, the epoch is 64, and the batch size is 8; in the significant target detection task, the pre-trained model needs to be loaded, AdamW is used as the optimizer, the initial learning rate is set to 0.0001, the epoch is 50, and the batch size is 10.

[0132] The loss function is used to evaluate the performance of the network model, and the loss functions defined for the reconstruction model and various task models are as follows:

[0133] For the training of the reconstruction model, the following mean square error loss function is used:

[0134]

[0135] Where N is the number of samples, y i is the true value of the i-th sample, is the predicted value of the i-th sample, the mean square error loss function calculates the square of the difference between the predicted value and the true value of each sample, and then averages the square errors of all samples;

[0136] For the training of the complete Token adaptive optimization image classification visual large model, the following cross-entropy loss function is used:

[0137] L=-log(p y )

[0138]

[0139] wherein P y is the probability that the model predicts the sample belongs to the true class y, by taking the negative logarithm to quantify the relationship between the probability of predicting the correct class and the loss, the lower the probability, the greater the loss; in P y the formula, z y is the output score of the model for the true class y, z j is the output score for all C classes, this formula is essentially a softmax function, which converts the original output score of the full token adaptive optimization image classification visual large model into a probability distribution, so that the sum of the probabilities of all classes is 1.

[0140] For the instance segmentation task, the following pixel-wise cross-entropy loss function is used:

[0141]

[0142] wherein N is the number of pixels in the image, each pixel has C classes, and the model is used to predict the probability distribution q i =(q i1 ,q 12 ,…,q iC ) of each pixel belonging to each class, wherein i=1,2,3,...,N, and the true label p i is a one-hot vector representing the true class of the pixel, and the total loss of an image is the average of all pixel losses.

[0143] For the salient object detection task, the following binary cross-entropy loss function is used:

[0144]

[0145] wherein N is the number of pixels in the image, y is the true label of the sample, and takes the value of 0 or 1. When y=1, it means that the sample belongs to the positive class; when y=0, it means that the sample belongs to the negative class. is the probability that the model predicts the sample belongs to the positive class, and the value range is [0,1].

[0146] Step five, based on the pre-processed complete token adaptive optimization visual large model test set in step four, using the complete token adaptive optimization visual large model and the weight file of the image classification task, the weight file of the instance segmentation task and the weight file of the salient object detection task trained in step four, through the complete token adaptive optimization visual large model constructed in step three, the model inference is carried out, and the final image classification result, instance segmentation result image and salient object detection result image are obtained.

[0147] 5.1, load the complete Token adaptive optimization visual large model structure, read the model weight file corresponding to the structure after complete training iteration, and load it into the complete Token adaptive optimization visual large model structure, set the complete Token adaptive optimization visual large model as an inference model, and fix the parameters of each network structure; the complete inference flowchart is shown in Figure 3

[0148] 5.2, the test set of the complete Token adaptive optimization visual large model includes image classification test set, instance segmentation test set and saliency target detection test set, and the image classification test set, instance segmentation test set and saliency target detection test set are respectively input into the corresponding complete Token adaptive optimization visual large model. The complete Token adaptive optimization visual large model first realizes adaptive selection of key information blocks in the image by using the Token adaptive selection module to obtain Token position coordinates with large information amount; secondly, the self-attention optimization module is used to adaptively adjust the attention weight according to the information amount of each image block, so as to ensure that more computing resources and attention are concentrated in the area with large information amount in the training process, and the area with small information amount is not concerned, and the features are accurately extracted; finally, the extracted image classification dataset features, instance segmentation dataset features and saliency target detection dataset features are input into the corresponding output head according to the specific task in the downstream task output module, and the corresponding output result is obtained, that is, the image classification result image, the instance segmentation result image and the saliency target detection result image.

[0149] 5.3, compare the complete Token adaptive optimization image classification visual large model with three methods ViT, Swin and T2T in image classification task, the test set is CIFAR10, CIFAR100 and Flowers102 dataset, and the Top1 accuracy is used to judge the model result, the larger the value, the more accurate the model classification result.

[0150] Table 1 Comparison of accuracy of different methods

[0151]

[0152] ​ViT, Swin, T2T are traditional image classification visual large models, as shown in Table 1, the classification accuracy of ViT on CIFAR10, CIFAR100 and Flowers102 data sets is 76.66%, 47.11%, 29.12% respectively, the classification accuracy of Swin on CIFAR10, CIFAR100 and Flowers102 data sets is 59.47%, 53.28%, 34.51% respectively, the classification accuracy of T2T on CIFAR10, CIFAR100 and Flowers102 data sets is 80.12%, 51.78%, 31.73% respectively; and the classification accuracy of the complete Token self-adaptive optimization image classification visual large model constructed by the application on the CIFAR10 data set is 20.39% higher than that of Swin, the classification accuracy on the CIFAR100 data set is 2.18% higher than that of ViT, and the classification accuracy on the Flowers102 data set is 23.59% higher than that of T2T; in the remaining comparative experiments, the classification accuracy is also higher than that of ViT, Swin and T2T three methods.

[0153] The application also provides a visual large model Token self-adaptive optimization system based on differential evolution, comprising:

[0154] The data processing module is used for realizing data processing on the image classification data set, the instance segmentation data set and the saliency object detection data set in step one, obtaining all Tokens corresponding to each image through the Patch Embedding and position coding method; obtaining multiple groups of Tokens corresponding to each image through a random selection method, and outputting all Tokens corresponding to each image and multiple groups of Tokens selected from each image by the data processing;

[0155] The first module is used for realizing the construction of the Token self-adaptive selection module, the self-attention optimization module and the downstream task output module in step two; the Token self-adaptive selection module is realized by a differential evolution algorithm, and the differential evolution algorithm comprises a reconstruction model; the self-attention optimization module is a network model based on Transformer; and the downstream task output module comprises an image classification task output head, an instance segmentation task output head and a saliency object detection task output head;

[0156] The model construction module is used for realizing the construction of the Token self-adaptive selection module, the self-attention optimization module and the downstream task output module based on step two in step three, and constructing a complete Token self-adaptive optimization visual large model;

[0157] The model training module is used to realize the following steps: obtaining the reconstruction data set to be trained in step four, performing data preprocessing on the reconstruction data set to be trained, obtaining the training set of the reconstruction model, inputting the training set of the reconstruction model into the reconstruction model in the Token adaptive selection module constructed in step two, reconstructing the original image, calculating the loss between the reconstructed original image and the real image, updating the parameters through back propagation for training, and obtaining the corresponding weight file; obtaining the downstream task data set to be trained, performing data preprocessing and division on the downstream task data set to be trained, obtaining the training set and test set of the complete Token adaptive optimization visual large model, inputting the training set of the complete Token adaptive optimization visual large model into the complete Token adaptive optimization visual large model constructed in step three, calculating the loss between the training result of the model and the real data, updating the parameters through back propagation for training, obtaining the trained complete Token adaptive optimization visual large model and the weight file of the image classification task, the weight file of the instance segmentation task and the weight file of the saliency target detection task, constructing the complete Token adaptive optimization visual large model, and outputting the results of the image classification task, the instance segmentation task and the saliency target detection task according to the corresponding weight file.

[0158] The model inference module is used to realize the following steps: based on the test set of the complete Token adaptive optimization visual large model after preprocessing in step four, using the complete Token adaptive optimization visual large model trained in step four and the weight file of the image classification task, the weight file of the instance segmentation task and the weight file of the saliency target detection task, performing model inference through the complete Token adaptive optimization visual large model constructed in step three, and obtaining the final image classification result, the instance segmentation result image and the saliency target detection result image.

[0159] The application also provides a visual large model Token adaptive optimization device based on differential evolution, which comprises:

[0160] The memory stores the computer program of the visual large model Token adaptive optimization method based on differential evolution, and is a computer-readable device.

[0161] The processor is used to execute the computer program to realize the visual large model Token adaptive optimization method based on differential evolution.

[0162] The application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the visual large model Token adaptive optimization method based on differential evolution can be realized.

Claims

1. A visual large model token self-adaptive optimization method based on differential evolution, characterized in that, The method comprises the following steps: Step one, data processing is performed on the image classification dataset, instance segmentation dataset and saliency object detection dataset, all Tokens corresponding to each image are obtained through the method of Patch Embedding and position coding, and multiple groups of Tokens corresponding to each image are obtained through random selection, and all Tokens corresponding to each image and the multiple groups of Tokens selected from each image are outputted; Step two, a Token adaptive selection module, a self-attention optimization module and a downstream task output module are constructed; the Token adaptive selection module is realized by a differential evolution algorithm, the differential evolution algorithm comprises a reconstruction model; the self-attention optimization module is a network model based on a Transformer; and the downstream task output module comprises an image classification task output head, an instance segmentation task output head and a saliency object detection task output head; Step three, based on the Token adaptive selection module, the self-attention optimization module and the downstream task output module constructed in step two, a complete Token adaptive optimization visual large model is constructed; Step four, a reconstruction dataset to be trained is obtained, data preprocessing is performed on the reconstruction dataset to be trained, a training set of the reconstruction model is obtained, the training set of the reconstruction model is inputted into the reconstruction model in the Token adaptive selection module constructed in step two to reconstruct an original image, the reconstructed original image and the real image are used to calculate a loss, the parameters are updated through back propagation for training, and a corresponding weight file is obtained; a downstream task dataset to be trained is obtained, data preprocessing and division are performed on the downstream task dataset to be trained, a training set and a test set of the complete Token adaptive optimization visual large model are obtained, the training set of the complete Token adaptive optimization visual large model is inputted into the complete Token adaptive optimization visual large model constructed in step three, the training result of the model is used to calculate a loss with the real data, the parameters are updated through back propagation for training, and a trained complete Token adaptive optimization visual large model and weight files of an image classification task, an instance segmentation task and a saliency object detection task are obtained, the complete Token adaptive optimization visual large model is constructed, and the results of the image classification task, the instance segmentation task and the saliency object detection task are outputted according to the corresponding weight files; Step five, based on the test set of the complete Token adaptive optimization visual large model after preprocessing in step four, the complete Token adaptive optimization visual large model trained in step four and the weight files of the image classification task, the instance segmentation task and the saliency object detection task are used to perform model reasoning through the complete Token adaptive optimization visual large model constructed in step three, and finally, image classification results, instance segmentation result images and saliency object detection result images are obtained.

2. The visual large model token adaptive optimization method based on differential evolution according to claim 1, characterized in that, The specific method of step one comprises: Step 1.1, the image classification dataset, instance segmentation dataset and salient object detection dataset are divided into fixed size image blocks by Patch Embedding, and each image block is flattened into a vector and mapped to a high dimensional space by linear transformation to form a serialized embedding representation; Step 1.2, add position encoding to the embedding representation of each image block, the dimension of the position encoding is the same as the dimension of the image block embedding, the embedding representation of each image block is added to the corresponding position encoding to obtain the corresponding Tokens; randomly select multiple groups of Tokens from all Tokens, and the number of selected Tokens in each group is one quarter of the total number of Tokens in the image.

3. The visual large model token adaptive optimization method based on differential evolution according to claim 1, characterized in that, The specific method of the step two comprises: Step 2.1, the Token adaptive selection module is realized by a differential evolution algorithm, and the differential evolution algorithm includes a reconstruction model as the objective function in the differential evolution algorithm; the Token adaptive selection module is used to find the Tokens coordinates with large information quantity in the image, and the input of the Token adaptive selection module is the multiple groups of Tokens obtained by the data processing in step one, and the output is the coordinates of the group of Tokens with large information quantity; The flow of the differential evolution algorithm is as follows: The input of the differential evolution algorithm is multiple groups of Tokens, and the output is the coordinates of the group of Tokens with the largest information quantity; each group of Tokens is input into the reconstruction model of the differential evolution algorithm, and multiple reconstruction tensors are generated by processing multiple groups of Tokens, each reconstruction tensor matches the original input tensor in dimension, the loss function between each reconstruction tensor and the original tensor is calculated to generate multiple loss values; wherein each group of Tokens is associated with its respective loss value; Based on the coordinates of the group of Tokens with the smallest loss value, mutation operation is performed on the coordinates of the multiple groups of Tokens, crossover operation is performed by exchanging the coordinates of Tokens from different groups to obtain new coordinates of multiple groups of Tokens, and new multiple groups of Tokens are obtained according to the new coordinates of multiple groups of Tokens; the obtained new multiple groups of Tokens are input into the reconstruction model again for image reconstruction, and a new multiple loss values are calculated by the objective function; the new loss value obtained is compared with the loss value of the last round, the group of Tokens with the smallest loss value is selected, and the mutation, crossover, loss value calculation by the objective function and selection operations are iterated for N times to finally select the group of Tokens with the smallest loss value, and the coordinates thereof are output to obtain the Tokens coordinates with large information quantity; The reconstruction model is composed of an encoder containing 6 Transformers and a decoder composed of 3 convolutional layers, the encoder is responsible for feature extraction, and the decoder is used to reconstruct the original image; each encoder is composed of a SelfAttention layer, two Layer Norm layers, and a Feed Forward layer, the output of the Self Attention layer is connected to the input of the first Layer Norm layer, the output of the first Layer Norm layer is connected to the input of the Feed Forward layer, and the output of the Feed Forward layer is connected to the input of the second Layer Norm layer; the 3 convolutional layers of the decoder are connected in turn; In the reconstruction model, after each group of Tokens is processed by the encoder, it is placed in the corresponding position of the tensor with the same width and height as the original image according to the coordinates, and the remaining positions are filled with Gaussian noise, and then the image is recovered by the decoder; output the reconstruction tensor corresponding to each group of Tokens; Step 2.2, the self-attention optimization module is a network model based on Transformer, which is used to extract image features, the input of the self-attention optimization module is the Tokens coordinates with large information amount identified by the Token adaptive selection module in step 2.1 and all Tokens processed in step one, and the output of the self-attention optimization module is the image classification dataset features, instance segmentation dataset features, and saliency object detection dataset features; The self-attention optimization module is a network model based on Transformer, which contains multiple encoders, each encoder is composed of a SelfAttention layer, two Layer Norm layers, a Feed Forward layer, and an MLP layer, the output of the SelfAttention layer is connected to the input of the first Layer Norm layer, the output of the first Layer Norm layer is connected to the input of the Feed Forward layer, the output of the Feed Forward layer is connected to the input of the second Layer Norm layer, and the output of the second Layer Norm layer is connected to the input of the MLP layer; The specific operation of the self-attention optimization module is as follows: The information-rich Token coordinates identified by the Token adaptive selection module in step 2.1 and all Tokens processed in step one are input into the Self Attention of the self-attention optimization module, and the Self Attention layer includes Evolutionary score and Similarity score; the Self Attention layer first assigns high scores to information-rich Tokens according to the information-rich Token coordinates, and assigns low scores to the remaining information-poor Tokens, to obtain the Evolutionary score of each Token; when calculating self-attention, first perform the query (Q) and key (K) operations, multiply the obtained Similarity score of each Token by the Evolutionary score of each Token as the new score of each Token, then normalize the modified score, thereby obtaining the final attention weight of each Token, and the adjusted final attention weight is then applied to the value (V) to complete the self-attention operation, and the Tokens after context perception are generated through the self-attention operation; the Tokens after context perception are input into the first Layer Norm layer for layer normalization to obtain normalized Tokens, and the normalized Tokens are input into the Feed Forward layer for nonlinear transformation to obtain nonlinearly transformed Tokens; the nonlinearly transformed Tokens are input into the second Layer Norm layer for secondary normalization to obtain second-normalized Tokens; the Tokens after secondary normalization are input into the MLP layer for dimension adjustment of the Tokens, and the MLP layer outputs image classification dataset features, instance segmentation dataset features, and saliency target detection dataset features; Step 2.3, the downstream task output module outputs the results of the image classification task, the instance segmentation task, and the saliency target detection task according to the image classification dataset features, the instance segmentation dataset features, and the saliency target detection dataset features output by the self-attention optimization module in step 2.2; The image classification dataset features, the instance segmentation dataset features, and the saliency target detection dataset features output by the self-attention optimization module in step 2.2 are input into the downstream task output module, and the downstream task output module outputs the class of the image, the instance segmented image, and the saliency target detected image according to the image classification task output head, the instance segmentation task output head, and the saliency target detection task output head, respectively; The implementation method of the image classification task is as follows: In the image classification task, the dataset is labeled, where each image corresponds to a corresponding class label, a single image data is read, the image is adjusted to a format suitable for the model requirements, and the three color channels of the image in the dataset are normalized. The image classification dataset features output by the self-attention optimization module in step 2.2 are converged into a class Token, which is mapped to a one-dimensional tensor through a multilayer perceptron, the length of the tensor is the same as the number of classes in the image classification task, and the value of each position in the tensor represents the probability of the image belonging to the corresponding class, completing the final image classification task; The implementation method of the instance segmentation task is: In the instance segmentation task, the image is adjusted to adapt to the format required by the model, and the three color channels of the image are normalized respectively; The instance segmentation dataset features output by the self-attention optimization module in step 2.2 are linearly mapped or mask transformer implemented point by point through the decoding process, and the training adopts the cross-entropy loss of each pixel for end-to-end optimization, and the instance segmentation dataset features are up-sampled, and the maximum index operation is applied to determine the unique class for each pixel, and finally the result image of the instance segmentation task is output; The implementation method of the salient object detection task is: In the salient object detection task, the image is adjusted to adapt to the format required by the model, and the salient object detection dataset features output by the self-attention optimization module in step 2.2 are sent into the decoder together with the hidden variable, and the initial value of the hidden variable comes from the standard normal distribution, and the decoder integrates the salient object detection dataset features and the hidden variable to generate a saliency map, which identifies the location and range of the salient object in the image.

4. The visual large model token adaptive optimization method based on differential evolution according to claim 1, characterized in that, The step three specifically includes: Step 3.1, input the multiple groups of Tokens obtained in step one into the Token adaptive selection module constructed in step 2.1, the image is divided into small blocks and converted into feature vector Tokens through patch embedding, then a part of Tokens are randomly selected as the initial population of the differential evolution algorithm, and the algorithm iterates with the trained reconstruction model as the objective function, the reconstruction model is composed of an encoder containing 6 Transformers and a decoder composed of 3 convolutional layers, the encoder is responsible for feature extraction, and the convolutional layers are used for reconstructing the original image; in the reconstruction model, the selected Tokens are processed by the encoder and placed in the corresponding position of the tensor with the same width and height as the original image according to the coordinates, the remaining positions are filled with Gaussian noise, and then the image is restored through the decoder; the differential evolution algorithm compares the differences between the reconstructed images of different groups of Tokens and the original image by calculating the mean square error to judge the reconstruction effect, realizes adaptive selection of Tokens, and finally outputs the corresponding coordinates of the Tokens with large information quantity in the original image; Step 3.2, the information of the Tokens coordinates output by the Token adaptive selection module constructed in step 2.1 and the all Tokens obtained by data processing in step one are input into a self-attention optimization module, the self-attention optimization module uses coordinate recognition to accurately distinguish important and unimportant Tokens, and according to an importance evaluation system, each Token is assigned a weight between 0 and 1 as an important weight value of each Token, at the same time, the self-attention learning calculates the correlation scores between Tokens, and the correlation scores are multiplied by the important weight, the product integrates the Token importance and the correlation relationship, finally the self-attention optimization module outputs the learned image classification dataset features, instance segmentation dataset features and saliency object detection dataset features through an MLP; Step 3.3, the image classification dataset features, instance segmentation dataset features and saliency object detection dataset features output by the self-attention optimization module constructed in step 2.2 are transmitted to a downstream task output module, and the downstream tasks include image classification task, instance segmentation task and saliency object detection task; in the image classification task, the probabilities of each category of the picture are calculated through a multi-layer perceptron, and finally the category with the maximum probability is output; in the instance segmentation task, based on an end-to-end optimization strategy, each pixel in the image is accurately determined to a unique category, and then an instance segmentation map is generated; and in the saliency object detection task, the saliency object detection dataset features output by the self-attention optimization module in step 2.2 are input into a decoder together with the latent variable to generate a saliency map.

5. The visual large model token adaptive optimization method based on differential evolution according to claim 1, characterized in that, The specific method of the step four preprocessing includes: reading the reconstruction dataset to be trained, adjusting the reconstruction dataset image to 3x224x224 and converting it into a format consistent with the reconstruction model, and performing normalization processing on the three color channels of the reconstruction dataset image respectively; reading the image classification dataset to be trained, adjusting the image classification dataset image to 3x224x224 and converting it into a format consistent with the complete Token adaptive optimization image classification visual large model, and performing normalization processing on the three color channels of the image classification dataset image respectively; reading the instance segmentation dataset to be trained, adjusting the instance segmentation dataset image to 3x480x480 and converting it into a format consistent with the complete Token adaptive optimization instance segmentation visual large model, and performing normalization processing on the three color channels of the instance segmentation dataset image respectively; reading the saliency object detection dataset to be trained, adjusting the saliency object detection dataset image to 3x384x384 and converting it into a format consistent with the complete Token adaptive optimization saliency object detection visual large model, and performing normalization processing on the three color channels of the saliency object detection dataset image respectively; The loss function is used to evaluate the performance of the network model, and the loss functions defined for the reconstruction model and the task model are as follows: For the training of the reconstruction model, the following mean square error loss function is used: where N is the number of samples, y i is the true value of the i-th sample, is the predicted value of the i-th sample, and the mean squared error loss function is used to calculate the square of the difference between the predicted value and the true value of each sample, and then average the squared errors of all samples. For the training of the complete Token adaptive optimization image classification visual large model, the following cross-entropy loss function is used: L = -log(p y ) where P y is the probability that the model predicts the sample belongs to the true class y, the relationship between the probability of predicting the correct class and the loss is quantified by taking the negative logarithm, the lower the probability, the greater the loss; in the formula P y , z y is the output score of the model for the true class y, z j is the output score for all C classes, this formula is essentially a softmax function, which converts the original output score of the full token adaptive optimization image classification visual large model into a probability distribution, so that the sum of the probabilities of all classes is 1; For the instance segmentation task, the following pixel-wise cross-entropy loss function is used: where N is the number of pixels in the image, each pixel has C classes, and the model is used to predict the probability distribution q of each pixel belonging to each class i = (q i1 , q 12 ,..., q iC ), where i = 1, 2, 3,..., N, and the real label p i is a one-hot vector representing the real class of the pixel, and the total loss of an image is the average of the loss of all pixels; For the salient object detection task, the following binary cross-entropy loss function is used: where N is the number of pixels in the image, y is the true label of the sample, and takes the value of 0 or 1; when y = 1, it means that the sample belongs to the positive class; when y = 0, it means that the sample belongs to the negative class; is the probability that the model predicts that the sample belongs to the positive class, and the value range is [0, 1].

6. The visual large model token adaptive optimization method based on differential evolution according to claim 1, characterized in that, The step five specifically includes: Step 5.1, loading the complete Token adaptive optimization visual large model structure, reading the model weight file corresponding to the structure after complete training iteration, and loading it into the complete Token adaptive optimization visual large model structure, setting the complete Token adaptive optimization visual large model as an inference model, and fixing the parameters of each network structure; Step 5.2, the test set of the complete Token adaptive optimization visual large model includes image classification test set, instance segmentation test set and salient object detection test set, the image classification test set, instance segmentation test set and salient object detection test set are respectively input into the corresponding complete Token adaptive optimization visual large model, the complete Token adaptive optimization visual large model first realizes the adaptive selection of key information blocks in the image by using the Token adaptive selection module, and obtains the Token position coordinates with large information amount; secondly, the self-attention optimization module is used to adaptively adjust the attention weight according to the information amount of each image block, so as to ensure that more calculation resources and attention are concentrated in the area with large information amount in the training process, and the area with small information amount is not concerned, and the features are accurately extracted; finally, the extracted image classification dataset features, instance segmentation dataset features and salient object detection dataset features are input into the corresponding output head according to the specific task in the downstream task output module, and the corresponding output result is obtained, that is, the image classification result image, the instance segmentation result image and the salient object detection result image; Step 5.3, comparing the complete Token adaptive optimization image classification visual large model with the image classification visual large model, the Top1 accuracy is used to judge the model result, and the larger the value is, the more accurate the model classification result is.

7. A differential evolution-based visual large model token adaption optimization system based on the method of any one of claims 1 to 6, characterized in that, It includes: The data processing module is used for data processing of image classification dataset, instance segmentation dataset and salient object detection dataset, and all Tokens corresponding to each image are obtained by Patch Embedding and position encoding method; a plurality of groups of Tokens corresponding to each image are obtained by random selection, and all Tokens corresponding to each image and a plurality of groups of Tokens selected from each image are output by the data processing; The first module is used for constructing the Token adaptive selection module, the self-attention optimization module and the downstream task output module; the Token adaptive selection module is realized by differential evolution algorithm, and the differential evolution algorithm includes a reconstruction model; the self-attention optimization module is a network model based on Transformer; the downstream task output module includes an image classification task output head, an instance segmentation task output head and a salient object detection task output head; The model construction module is used to implement the Token adaptive selection module, the self-attention optimization module and the downstream task output module to construct a complete Token adaptive optimization visual large model. The model training module is used to obtain a reconstruction data set to be trained, perform data preprocessing on the reconstruction data set to be trained, obtain a training set of the reconstruction model, input the training set of the reconstruction model into a reconstruction model in the Token adaptive selection module to reconstruct an original image, calculate a loss between the reconstructed original image and a real image, update parameters through back propagation for training, and obtain a corresponding weight file. The downstream task data set to be trained is obtained, data preprocessing and division are performed on the downstream task data set to be trained, a training set and a test set of the complete Token adaptive optimization visual large model are obtained, the training set of the complete Token adaptive optimization visual large model is input into the complete Token adaptive optimization visual large model, a training result of the model is calculated with real data to obtain a loss, parameters are updated through back propagation for training, and a trained complete Token adaptive optimization visual large model and a weight file of an image classification task, a weight file of an instance segmentation task and a weight file of a saliency object detection task are obtained. The model inference module is used to implement a test set based on the preprocessed complete Token adaptive optimization visual large model, utilize the trained complete Token adaptive optimization visual large model and the weight file of the image classification task, the weight file of the instance segmentation task and the weight file of the saliency object detection task, perform model inference through the complete Token adaptive optimization visual large model, and obtain final image classification results, instance segmentation result images and saliency object detection result images.

8. A device for optimizing visual large model Token adaptation based on differential evolution, characterized in that, It comprises: a memory for storing a computer program of the differential evolution-based visual large model Token adaptive optimization method according to any one of claims 1-6, which is readable by a computer; a processor for implementing the differential evolution-based visual large model Token adaptive optimization method when executing the computer program according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer program is stored in a computer-readable storage medium, and the computer program can implement the differential evolution-based visual large model Token adaptive optimization method according to any one of claims 1-6 when executed by a processor.

Images