Image classification method and device based on visual large model Token parallelization

Abstract

The invention discloses an image classification method and device based on visual large model Token parallelization, and relates to the technical field of image classification, and the method comprises the steps: inputting a to-be-processed image into a dynamic resolution predictor module to determine the optimal resolution of the to-be-processed image when a main node receives an image processing request, and carrying out the prediction of the optimal resolution of the to-be-processed image; adjusting the size of the to-be-processed image based on the optimal resolution to obtain a size-adjusted image; when the optimal resolution is greater than a preset resolution value, determining a dynamic planning result by using a dynamic planning algorithm according to the waiting queue and image processing requirements; packaging the first request in the waiting queue into a plurality of token packets; and according to a dynamic planning result, respectively sending all token packets to different edge devices for processing. According to the invention, the image token number optimization device and the reasoning service planning device are arranged, so that the response speed and stability of the edge device cluster during image classification can be improved.

Description

Technical Field

[0001] The present application relates to the field of image classification technology, and in particular to an image classification method and device based on the parallelization of visual large model tokens. Background Art

[0002] With the rapid development of IoT technology, the interconnectivity of smart devices is becoming increasingly prevalent. Leveraging the unique characteristics of IoT, various industries are rapidly leveraging this technology to enable seamless network connectivity among their devices, spurring numerous practical applications such as medical robots, smart wearables, and smart homes. According to relevant organizations, by 2025, the number of IoT devices connected to the global network will exceed 100 billion, generating an estimated 300 zettabytes of data. This surge in devices and data poses a significant challenge to traditional data processing models. Traditionally, data from edge devices is typically uploaded to the cloud for centralized processing, but this approach faces numerous challenges. For example, with the explosive growth in the number of devices, cloud platforms may be limited by network latency, excessive bandwidth requirements, and increased power consumption, resulting in reduced overall efficiency. Furthermore, large-scale data transmission places significant pressure on network infrastructure, increasing costs and potentially hindering applications with high real-time requirements. To address these challenges, edge computing technology has emerged. Edge computing moves data processing and services from the central cloud to the network edge, close to the data source, allowing edge devices to perform data processing and analysis locally. This architecture reduces network bandwidth requirements by reducing data transmission volume, significantly reducing network latency and power consumption. The advantage of edge computing lies in its ability to deliver intelligent services at the point of data generation, enabling localized processing and meeting real-time requirements. This enables more efficient applications such as autonomous driving, the Industrial Internet of Things, and smart cities. This technological transformation not only optimizes resource utilization but also drives the further development of the IoT ecosystem.

[0003] Deep learning has gradually become mainstream in various edge computing services, primarily due to significant breakthroughs in computer vision and natural language processing. In recent years, technologies such as deep neural networks (DNNs) and convolutional neural networks (CNNs) have demonstrated outstanding performance in these fields, enabling rapid inference and accurate results. In particular, CNNs have been widely used in tasks such as image recognition, object detection, and video analysis. Their powerful feature extraction capabilities enable them to complete inference tasks in a short time, even when running lightweight models on edge devices. In recent years, the performance of large-scale vision models based on the Transformer has gradually surpassed that of traditional convolutional neural networks (CNNs), with models such as the Vision Transformer (ViT) performing particularly well. Unlike CNNs, ViT relies on a unique self-attention mechanism, which enables it to capture global information within an image, resulting in superior performance in downstream tasks such as image classification. By modeling the entire image region, the self-attention mechanism effectively handles long-range dependencies, overcoming the bottleneck of CNNs being limited to a local receptive field.

[0004] However, ViT also has significant drawbacks. Its smallest computational unit is called a token, and the number of tokens is proportional to the resolution of the input image. As image resolution increases, ViT's inference complexity increases significantly. This is because the computational complexity of the self-attention mechanism quadratically scales with the number of tokens. Specifically, as the input image size increases and the number of tokens increases, the computational cost of self-attention also increases exponentially. This significantly increases the model's complexity and latency when inferring on high-resolution images. Therefore, while ViT offers performance advantages, its high demand for computing resources, especially inference efficiency issues in high-resolution scenarios, poses a major challenge in edge computing scenarios. In edge computing environments, computing resources and power consumption are often strictly constrained, and the high computational complexity of large visual models like ViT makes them challenging to deploy on edge devices. However, given the large number of edge devices and their tendency to form clusters, researchers have proposed several optimization strategies for deploying computationally intensive models like ViT on multiple edge devices to address the deployment challenges. 1) Model parallelism: The hierarchical structure of ViT is partitioned and assigned to different edge devices, with inference executed serially by these devices. Each device is responsible for computing a specific layer, and the results are passed to the next device in sequence. This effectively distributes the computational burden. 2) Tensor parallelism: Certain model parameters of ViT (such as the weight matrix) are assigned to different edge devices by column. Tensor information is continuously exchanged and transmitted between devices, and the entire inference process is performed in parallel on multiple devices. This reduces the memory overhead of a single device and improves inference speed. 3) Sequence parallelism: The token sequence input to ViT is distributed sequentially to different edge devices. Each device processes its assigned tokens and exchanges inference results to ensure the integration of global context information. This strategy can parallelize the processing of multiple tokens, thereby improving inference efficiency. Although these optimization strategies have alleviated the computational bottlenecks of models such as ViT on edge devices to a certain extent and completed inference tasks through multi-device collaboration, they also introduce new problems, particularly the frequent communication overhead between devices. Due to the distributed deployment of edge devices, data and intermediate results need to be frequently transmitted between devices during inference, which greatly increases network bandwidth requirements and leads to high end-to-end latency. Especially in poor network conditions or with frequent communications, this overhead can significantly impact the model's inference performance. Although computing tasks are shared, overall inference latency remains high. Fluctuating loads at the edge also pose challenges to the service system, making resource management and scheduling difficult, leading to latency fluctuations and decreased service quality. In such situations, optimizing the model's inference process is crucial. Effective load balancing and caching strategies are needed to reduce unnecessary communication overhead and improve the overall system's responsiveness and stability. Summary of the Invention

[0005] The purpose of this application is to provide an image classification method and device based on the parallelization of visual large model tokens, which can improve the response speed and stability of edge device clusters during image classification.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides an image classification method based on large visual model Token parallelization, wherein the image classification method based on large visual model Token parallelization is applied to an edge device cluster;

[0008] The master node of the edge device cluster is integrated with an image token quantity optimization device and an inference service planning device; the image token quantity optimization device includes: a dynamic resolution predictor module and a token heterogeneous packaging strategy module; multiple edge devices in the edge device cluster are integrated with a large visual model;

[0009] The image classification method based on the parallelization of visual large model Token includes:

[0010] Upon receiving an image processing request, the master node inputs the image to be processed into a dynamic resolution predictor module to determine an optimal resolution of the image to be processed, and resizes the image to be processed based on the optimal resolution to obtain a resized image; the image processing request is sent by a user; the image processing request includes an image processing requirement and a queue of images to be processed including a plurality of images to be processed;

[0011] When the optimal resolution is greater than the preset resolution value, the resized image is added to the waiting queue;

[0012] The inference service planning device uses a dynamic programming algorithm to determine a dynamic planning result based on the waiting queue and image processing requirements; the dynamic planning result includes the token distribution of all requests in the waiting queue and the edge device corresponding to each token packet;

[0013] The Token Heterogeneous Packaging Strategy Module packages the first request in the waiting queue into multiple token packages based on the resized image and the token distribution of the first request in the waiting queue.

[0014] According to the dynamic planning results, all token packets are sent to different edge devices for processing;

[0015] Obtain and integrate the calculation results of the large visual model in all multiple edge devices that receive the same label, and obtain the image classification result of the image to be processed corresponding to the first request in the waiting queue.

[0016] Optionally, the image processing requirements include a start time, a deadline and a utility.

[0017] Optionally, the image classification method based on visual large model Token parallelization further includes:

[0018] When the optimal resolution is less than or equal to the preset value, the resized image is sent to any idle edge device, and the calculation result of the visual large model in the edge device is obtained as the image classification result of the image to be processed.

[0019] Optionally, the dynamic resolution predictor module is a ResNet network;

[0020] The dynamic resolution predictor module is:

[0021]

[0022] represents the dynamic resolution predictor module; C x represents the predicted probability of resolution; Represent the prediction probabilities of the 1st, 2nd, and mth resolutions respectively;

[0023] In order to select the resolution with the highest probability, the one_hot function needs to be used;

[0024] The one_hot function is:

[0025] h=one_hot(argmax[g i +log C i ]);

[0026] The loss function when training the dynamic resolution predictor module is:

[0027]

[0028] in,

[0029] y i =f(X i );

[0030]

[0031] g i =-log(-logu)u~U(0,1);

[0032] Where L is the loss function when training the dynamic resolution predictor module; is the cross loss entropy; L ron is the first loss; η is the hyperparameter; η is the first loss Lron Impact factor; is the predicted label; y true is the true label; y i represents the output result of inputting the m-th resolution image into the visual large model; f(X i ) represents the visual large model; X i represents the m-th resolution image; E(*) is the mean function; F is the average GFLOPs at m resolutions; β is the GFLOPs target value; G max is the model GFLOPs at maximum resolution; G min h is the GFLOPs of the model with the minimum resolution; i is the corresponding prediction probability under the resolution image; G i is the model GFLOPs corresponding to each resolution; is Gaussian noise; τ is the temperature coefficient in the softmax function; is the calculation formula of e^x; u is a uniformly distributed random value; U(0,1) is uniformly distributed; g i is Gaussian noise; C i The value predicted for dynamic resolution.

[0033] Optionally, the large visual model comprises: a linear mapping layer, a transformer encoder, and a classification head network connected in sequence;

[0034] The workflow of the visual large model is as follows:

[0035] Divide the input image into multiple image blocks and determine the position code of each image block;

[0036] Input multiple image blocks into the linear mapping layer to obtain the token corresponding to each image block;

[0037] After combining multiple tokens with the position codes of the corresponding image blocks, a sequence of image block embedding vectors is formed;

[0038] Input the image block embedding vector sequence into the transformer encoder to obtain the image encoding;

[0039] The image encoding is input to the classification head network to obtain the classification result of the input image.

[0040] Optionally, the transformer encoder comprises a plurality of sequentially connected multi-head self-attention-multi-layer perceptron layers;

[0041] Any of the multi-head self-attention-multi-layer perceptron layers comprises a multi-head self-attention layer and a multi-layer perceptron connected in sequence;

[0042] The multi-head self-attention layer includes a normalization layer of residual connections and a multi-head self-attention mechanism;

[0043] The multi-layer perceptron includes a normalization layer with residual connections and a multi-layer perceptron.

[0044] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned image classification method based on the parallelization of visual large model tokens.

[0045] In a third aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image classification method based on the parallelization of the visual large model Token is implemented.

[0046] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned image classification method based on parallelization of visual large model Tokens.

[0047] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0048] This application provides an image classification method and device based on token parallelization of a large visual model. By introducing a dynamic resolution predictor and a token packaging method, high-resolution image inference can be parallelized on edge devices, thereby optimizing the cost and resource utilization of edge inference systems. By reducing the exchange of information and data transmission between edge devices, end-to-end inference latency is significantly reduced while maintaining a high level of inference accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is a flow chart of an image classification method based on visual large model Token parallelization in one embodiment of the present application;

[0051] Figure 2 This is a diagram showing the principle of image classification using the parallelization of the visual large model Token in one embodiment of the present application;

[0052] Figure 3 This is a schematic diagram of a large visual model in one embodiment of the present application;

[0053] Figure 4 This is a ViT parallel reasoning flowchart after inserting the token packing strategy in one embodiment of the present application;

[0054] Figure 5 This is the first flow chart of the token packaging strategy in one embodiment of the present application;

[0055] Figure 6 This is the second flow chart of the token packaging strategy in one embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0057] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0058] As the capabilities of large visual models continue to improve, their inference services on edge devices have shown great potential. The following are some important application scenarios:

[0059] Intelligent security monitoring: Edge devices can deploy large visual models to process surveillance video in real time, performing tasks such as anomaly detection, identity recognition, and behavioral analysis. By performing inference locally, the bandwidth pressure of uploading large amounts of video data to the cloud is reduced, while enabling faster responses and improving security.

[0060] Autonomous driving: In self-driving cars, large visual models can be used to identify dynamic environmental information such as traffic signs, pedestrians, and vehicles. Because autonomous driving requires extremely high real-time performance, deploying models on edge devices (such as onboard computing units) for real-time inference can significantly reduce latency and ensure timely and safe driving decisions.

[0061] Industrial Automation: Edge devices combined with large visual models can be applied in manufacturing for scenarios such as quality inspection, equipment failure prediction, and process monitoring. Intelligent reasoning on local factory equipment can improve inspection accuracy, reduce downtime caused by network latency, and ensure continuous production line operation.

[0062] Smart healthcare: Edge devices can use large visual models to perform local analysis of medical images (such as X-rays and CT scans) and generate diagnostic recommendations in real time. This local reasoning service not only improves data privacy but also reduces reliance on cloud resources, making medical devices more intelligent.

[0063] Smart Retail: Edge devices can use large visual models in retail scenarios to analyze customer behavior, manage inventory, and perform intelligent checkout. Real-time reasoning optimizes the customer experience, improves operational efficiency, and provides businesses with valuable operational data.

[0064] Drones and Robots: Large visual models on edge devices enable drones and robots to achieve autonomous perception and decision-making in complex environments. Drones can be used for environmental monitoring, logistics distribution, and disaster assessment, while robots can be applied to tasks such as warehouse management and logistics sorting.

[0065] These application scenarios demonstrate that as the deployment of large visual models on edge devices matures, their potential applications will continue to expand, encompassing fields such as intelligent transportation, smart manufacturing, healthcare, and retail services. Edge inference services not only improve real-time performance, reduce data transmission and network latency, but also enhance data privacy, significantly advancing the intelligent advancement of various industries.

[0066] In an exemplary embodiment, Figure 1 As shown, an image classification method based on large visual model Token parallelization is provided, and the image classification method based on large visual model Token parallelization is applied to an edge device cluster.

[0067] The master node of the edge device cluster integrates an image token quantity optimization device and an inference service planning device. The image token quantity optimization device includes a dynamic resolution predictor module and a token heterogeneous packaging strategy module. Multiple edge devices in the edge device cluster are integrated with a large visual model.

[0068] This efficient edge computing device, based on token parallelization of large visual models, is primarily deployed in an edge device cluster. Both the image token quantity optimization device and the inference service planning device are deployed on the edge device cluster's master node, providing strong scheduling capabilities. The edge device cluster's master node accepts user requests for high-resolution image classification and transmits them to the image token quantity optimization device. It also receives request information, such as start time, deadline, and utility, and passes this information to the inference service planning device. The image token quantity optimization device responds to user requests for high-resolution image processing. First, it uses a dynamic resolution predictor to reduce the number of tokens at the image level. For images that still maintain high resolution, the device leverages a heterogeneous token packaging strategy to achieve parallel token inference. The inference service planning device receives the high-resolution image and user request information from the image token quantity optimization device. Based on the waiting queue, it uses a dynamic programming algorithm to achieve maximum utility planning and proposes a token distribution for all requests in the waiting queue. Furthermore, the heterogeneous packaging strategy employed by the image token quantity optimization device is combined to dispatch heterogeneous token quantities to multiple edge devices, ultimately completing image classification through distributed and integrated inference.

[0069] Image classification methods based on Token parallelization of large visual models include:

[0070] Step 101: Upon receiving an image processing request, the master node inputs the image to be processed into the dynamic resolution predictor module to determine the optimal resolution for the image to be processed. The master node then resizes the image to be processed based on the optimal resolution, generating a resized image. An image processing request is sent by a user. It includes an image processing requirement and a queue of images to be processed, which includes multiple images to be processed. The image processing requirement includes a start time, a deadline, and a utility.

[0071] Step 102: When the optimal resolution is greater than the preset resolution value, the resized image is added to a waiting queue.

[0072] Step 103: The inference service planning device uses a dynamic programming algorithm to determine a dynamic planning result based on the waiting queue and image processing requirements. The dynamic planning result includes the token distribution of all requests in the waiting queue and the edge device corresponding to each token packet.

[0073] Step 104: The Token heterogeneous packaging strategy module packages the first request in the waiting queue into multiple token packages based on the resized image and the token distribution of the first request in the waiting queue.

[0074] Step 105: According to the dynamic planning results, all token packets are sent to different edge devices for processing.

[0075] Step 106: Obtain and integrate the calculation results of the visual large model in all multiple edge devices that receive the same label, and obtain the image classification result of the image to be processed corresponding to the first request in the waiting queue.

[0076] Step 107: When the optimal resolution is less than or equal to the preset value, the resized image is sent to any idle edge device, and the calculation result of the visual large model in the edge device is obtained as the image classification result of the image to be processed.

[0077] The dynamic resolution predictor module is a ResNet network.

[0078] The loss function when training the dynamic resolution predictor module is:

[0079] The dynamic resolution predictor module is:

[0080]

[0081] represents the dynamic resolution predictor module; C x represents the predicted probability of resolution; Represent the prediction probabilities of the 1st, 2nd, and mth resolutions respectively;

[0082] In order to select the resolution with the highest probability, the one_hot function needs to be used;

[0083] The one_hot function is:

[0084] h=one_hot(argmax[g i +log C i ]);

[0085] The loss function when training the dynamic resolution predictor module is:

[0086]

[0087] in,

[0088] y i =f(X i );

[0089]

[0090] This is to solve the invisible micro problem caused by the argmax function

[0091]

[0092] g i =-log(-logu)u~U(0,1);

[0093] Where L is the loss function when training the dynamic resolution predictor module; is the cross loss entropy; L ron is the first loss; η is the hyperparameter; η is the first loss L ron Impact factor; is the predicted label; y true is the true label; y i represents the output result of inputting the m-th resolution image into the visual large model; f(X i ) represents the visual large model; X i represents the m-th resolution image; E(*) is the mean function; F is the average GFLOPs at m resolutions; β is the GFLOPs target value; G max is the model GFLOPs at maximum resolution; G min h is the GFLOPs of the model with the minimum resolution; i is the corresponding prediction probability under the resolution image; G i The model GFLOPs corresponding to each resolution; is Gaussian noise; τ is the temperature coefficient in the softmax function; is the calculation formula of e^x; u is a uniformly distributed random value; U(0,1) is uniformly distributed. The large visual model includes: linear mapping layers, transformer encoders and classification head networks connected in sequence; g i is Gaussian noise; C i The value predicted for dynamic resolution.

[0094] The workflow of the visual large model is:

[0095] The input image is divided into multiple image blocks and the position encoding of each image block is determined.

[0096] Input multiple image blocks into the linear mapping layer to obtain the token corresponding to each image block.

[0097] After combining multiple tokens with the position encoding of the corresponding image blocks, an image block embedding vector sequence is formed.

[0098] The image block embedding vector sequence is input into the transformer encoder to obtain the image encoding.

[0099] The image encoding is input to the classification head network to obtain the classification result of the input image.

[0100] The transformer encoder consists of multiple sequentially connected multi-head self-attention-multilayer perceptron layers.

[0101] Any multi-head self-attention-multi-layer perceptron layer consists of a multi-head self-attention layer and a multi-layer perceptron connected in sequence.

[0102] The multi-head self-attention layer includes a normalization layer of residual connections and a multi-head self-attention mechanism.

[0103] The multi-layer perceptron includes a residual-connected Norm (normalization layer) and a multi-layer perceptron.

[0104] This application focuses on token reduction technology and inference service dynamic planning technology, and designs an image token quantity optimization device and inference service planning device adapted for edge devices. The main features include a ViT-based token packaging strategy, token parallelization inference, a dynamic resolution predictor, and an inference service dynamic planning algorithm. The image token quantity optimization device includes a dynamic resolution predictor module and a token heterogeneous packaging strategy module.

[0105] The image token quantity optimization device in this application is deployed in the master node of the edge device cluster, which is specifically used to process high-resolution image classification requests submitted by users, reduce them to the optimal size using a dynamic resolution predictor, and parallelize reasoning through a token packaging algorithm to reduce the reasoning complexity of each edge device. Figure 2 As shown, the dynamic resolution predictor network is applied after the high-resolution image is input and before the ViT network is input, and the token packing strategy is applied before ViT reasoning. At the same time, this application will allocate different numbers of tokens to different edge devices based on factors such as the results of the dynamic resolution predictor and the number of current service requests, and finally complete parallel reasoning in the edge device equipped with the Vision Transformer network.

[0106] In view of the differences in the difficulty of different images in the ViT reasoning process, this application designs a dynamic resolution predictor to optimize reasoning performance and resource utilization. The reasoning difficulty of an image is usually closely related to the complexity of its foreground objects and background information. For "difficult" images where the foreground and background information are more complexly entangled, the model faces greater challenges in extracting visual information, so the reasoning difficulty is higher and the confidence is often lower. In order to improve the accuracy of reasoning, such images are more suitable for reasoning using high resolution to capture more details and information. On the contrary, for "easy" images with simpler visual content, the foreground objects and background are clearly distinguished, and the visual features are not complex. The model can also correctly reason at a lower resolution. Such images do not require high resolution to obtain accurate results. Using a lower resolution for reasoning can effectively save computing resources and reduce reasoning time. First, this application uses the ResNet network as a resolution predictor network, which can predict a resolution with the highest ViT reasoning accuracy for each high-resolution image and change the input size of its image. This application represents the network as a prediction function P(·). For the input image x, this application designs multiple candidate resolution vectors in Represents the possibility of each of the m candidate resolutions, with the resolution having the highest probability possibility, but in order to obtain a discrete mask vector, the argmax function needs to be used:

[0107]

[0108] However, the argmax function can cause non-differentiable problems, so this application introduces the Gumble Softmax module to solve the non-differentiable problem during training:

[0109] g j =-log(-logu) u~U(0,1) (2)

[0110]

[0111] During training, the predictor is trained end-to-end together with ViT, and the image x is reduced to X1, X2, ..., X at m resolutions. m , output Aggregated into

[0112]

[0113] where y i =f(X i ) is the output result of ViT. This operation enables the model to learn the correct results at different resolutions. This application uses the model complexity GFLOPs regularization term to guide the learning of the predictor. First, use hi Calculate the average GFLOPs at m resolutions as Secondly, this application normalizes the absolute difference between the actual value F of GFLOPs and the target value β as the first loss value And the cross loss entropy of the true label and the predicted label is taken as the final loss function:

[0114]

[0115] The above process is the training process of the dynamic predictor. During the inference process, the dynamic resolution predictor selects the optimal resolution for the image. If the resolution is relatively small, the master node dispatches these low-resolution images to idle edge devices for inference. The inference time is significantly reduced compared to the original image. The dynamic resolution predictor considers difficult images (still high-resolution images after resizing) to require the introduction of the following token packaging strategy and parallel inference processing to reduce their inference time while ensuring inference accuracy.

[0116] After using the dynamic prediction network (ResNet) to predict the resolution of the image and adjust the size, this application needs to use ViT to perform reasoning analysis on the image. For the original ViT network that plays the role of reasoning, such as Figure 3 As shown in the figure, ViT divides the input image into multiple 16×16 pixel blocks (patches), then projects each patch into a fixed-length token embedding (token embedding), which is then fed into the self-attention network (transformer). The subsequent encoder operates on all block embeddings and finally selects the first category token (token) to input into the classification head to obtain the final inference result (image category).

[0117] Specifically, ViT converts a two-dimensional image The input is divided into a sequence of blocks with a resolution of P×P Where (H, W) is the resolution of the original image, C is the number of channels, is the number of blocks after segmentation.

[0118] All blocks are passed through the linear mapping layer to obtain tokens with an embedding length of D and a sequence length of N.

[0119] Then, a learnable token is added to the front of the token sequence, and then position embedding is added to each token. The resulting token sequence is used as the input z0 of the encoder.

[0120] like Figure 3As shown, the encoder consists of L groups of alternating multi-head self-attention layers MSA and multi-layer perceptrons MLP. Layer norm is applied before each block and residual connection is applied after each block. The multi-layer perceptron contains two linear layers and applies the GELU activation function. The forward process formula of ViT is:

[0121]

[0122] z′ l =MSA(LN(z l-1 ))+z l-1 ,l=1…L (7)

[0123] z l =MLP(LN(z′ l ))+z′ l ,l=1…L (8)

[0124]

[0125] In the calculation of the multi-head self-attention mechanism, the calculation of a single head is as follows:

[0126]

[0127] Attention(α,V)=αV(11)

[0128] Among them, Q, K and V represent the query matrix, key matrix and key-value matrix, Represents the self-attention score matrix between different tokens, and also includes class tokens ([CLS]token), which is also an important basis for the token packaging algorithm in this application.

[0129] For high-resolution images, this application needs to introduce a Token heterogeneous packaging strategy to reduce the inference time and complexity of ViT deployed in edge devices. The Token heterogeneous packaging strategy module is as follows:

[0130] like Figure 5 ,In the ViT reasoning process, based on the self-attention score matrix in ViT, this application designed a token packaging strategy, as shown in Table 1.

[0131] Table 1 Token packaging strategy table

[0132]

[0133] Algorithm 1 uses an empty list of length K to implement the token packaging strategy.

[0134] ①First, the first row vector α in the self-attention score matrix is used[CLS] , use the K tokens with the largest scores as cluster centers and store their index Idx in [I1,I2,…I k ], the columns of these tokens and the [CLS] token in the self-attention score matrix are set to negative infinity to prevent them from being selected later.

[0135] ②Then the algorithm will gradually traverse [I1,I2,…I k ], and select the index with the highest score from the self-attention score matrix for clustering until the distribution vector Π=[π1,π2,…π k ] requirements.

[0136] ③ The final algorithm selects the corresponding token based on the re-sorted index. At this point, the tokens of the K clusters are packaged and the [CLS] token is concatenated with them to form a unique token. It has a very high internal correlation without affecting the mutual reasoning.

[0137] Reasoning service planning device:

[0138] The inference service planning device is also deployed in the main node of the edge device cluster. Its core task is to intelligently plan the number of tokens allocated to each edge device based on the deadline and utility requirements of the user's request. The device dynamically adjusts the number of tokens allocated to each device in combination with the user's inference request to achieve the best balance between inference time and accuracy. Specifically, when the pre-utility of the user's request is low and the deadline is short, the system will reduce the number of tokens allocated to each device, thereby reducing the inference calculation amount and the maximum inference time. Although this may lead to a decrease in accuracy, it can meet the needs of fast response. On the contrary, for requests with higher accuracy requirements and longer deadlines, the system will increase the inference time and allocate more tokens to a certain edge device to improve the accuracy of the inference results. For this planning method,

[0139] This application sets the following optimization problem:

[0140]

[0141] Formula (14) obtains the maximum utility value by adjusting the token quantity distribution vector Πq of each user request. Formula (15) ensures that each request will not exceed the user's deadline. Formula (16) ensures that the sum of the token quantity distribution vectors does not exceed 1. Formula (17) ensures that the next request will not be processed in advance. This problem is obviously an NP-hard problem.

[0142] Therefore, this application introduces a dynamic programming method to dynamically plan the processing method of different user requests, and assign different token packaging strategies to each image. The specific algorithm is shown in Table 2 and Figure 6 shown.

[0143] Table 2 Token packaging strategy table

[0144]

[0145]

[0146] ① In this algorithm, first reorder the queue Q according to the deadline continuous time, so that the ones with earlier deadlines are processed in advance, and use t pre Discretize continuous time to better implement dynamic programming,

[0147] ② Simultaneous use and To assist in the implementation of the dynamic programming algorithm, where dp represents the cumulative utility at the current time, and C records the token distribution scheme Π of each request at the current time.

[0148] ③ During the iterative process of dynamic programming, this application mainly uses offline files to record the inference delay time and utility of each token distribution method in the token distribution method list List(Π). The utility is mainly measured based on the accuracy of different data sets. If the accuracy of ViT in a certain data set is relatively high, then the utility of the request of this data set is relatively low, and vice versa. By accumulating the utility of each solution into the dp matrix, this application can evaluate the effect of different distribution solutions in each iteration. Through the dynamic programming algorithm, this application can eventually obtain the corresponding dp and C matrices in the request queue.

[0149] ④ Finally, based on the C matrix obtained in ③, by analyzing all requests in the request queue Q, the algorithm can determine the optimal token scheduling scheme Π for each image in the queue, thereby achieving the maximum total utility value. Finally, with a different Π for each request, the edge master node implements the heterogeneous token packaging strategy mentioned in step 1, distributing different tokens to the edge device cluster for parallel inference.

[0150] According to the token scheduling scheme Π obtained in ②, perform parallel reasoning of the packaging strategy:

[0151] The specific process is: distribute the packaged tokens to K edge devices for parallel reasoning. Each edge device has the same ViT encoder parameters, and its structure is as follows: Figure 4 As shown, this application finally uses the idea of integrated reasoning to aggregate the reasoning results of each edge device to obtain the final output:

[0152]

[0153] The ViT model with token packaging strategy inserted optimizes the inference performance of edge device clusters by introducing a token parallelization mechanism.

[0154] In this model, this application first uses the first-layer encoder of ViT to generate a self-attention score matrix, which reflects the correlation between each token in the image.

[0155] Next, through the heterogeneous packaging strategy in Algorithm 1, this application dynamically divides these tokens into K packages. This heterogeneous packaging strategy takes into account the characteristics of each token and its importance in reasoning, ensuring that the computing load is evenly distributed among multiple edge devices. Each token package is assigned to a different edge device, which is equipped with an 11-layer encoder network (TransformerBlock), and then performs parallel reasoning. Each device independently completes the reasoning task of the assigned token. Finally, the reasoning results of each edge device will be returned to the master node, and the master node will integrate these results to generate the final classification result.

[0156] This application designs an image token quantity optimization device that uses a dynamic resolution predictor to effectively reduce the number of tokens for high-resolution images. Based on this, a new token packing strategy is proposed for images that still maintain high resolution. Compared with traditional model parallelism, tensor parallelism, and sequence parallelism, this strategy avoids information exchange between multiple edge devices, significantly reducing end-to-end inference latency while maintaining image classification inference accuracy. This application also designs an inference service planning device that uses a dynamic programming algorithm to combine user request information with the waiting queue to perform inference service planning, thereby obtaining the optimal utility solution for the current waiting queue. This device, combined with the heterogeneous packing strategy in the image token quantity optimization device, enables parallel inference. It successfully achieves maximum utility within a limited time, significantly reducing the impact of fluctuating loads and effectively addressing performance bottlenecks that may arise under high load conditions. The application of a dynamic programming algorithm ensures optimal resource allocation, enabling rapid response to user requests. This application utilizes an edge computing architecture to provide users with efficient inference services. By designing an image token quantity optimization device and an inference service planning device, this application achieves higher efficiency in data processing and transmission. This design significantly reduces end-to-end inference latency, providing a smoother user experience. Furthermore, dynamic adjustments to token distribution successfully mitigate the impact of load fluctuations on the service system, further improving service stability and reliability. This comprehensive solution provides effective technical support for inference services in edge computing environments, adapting to evolving user needs.

[0157] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an image classification method based on the parallelization of a large visual model Token is implemented.

[0158] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0159] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0161] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0162] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0163] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of this application. In summary, the contents of this specification should not be construed as limiting this application.

Claims

1. An image classification method based on the parallelization of visual large model tokens, characterized in that: The image classification method based on the parallelization of large visual model tokens is applied to an edge device cluster; The master node of the edge device cluster is integrated with an image token quantity optimization device and an inference service planning device; The image token quantity optimization device includes: a dynamic resolution predictor module and a token heterogeneous packaging strategy module; multiple edge devices in the edge device cluster are integrated with a large visual model; The image classification method based on the parallelization of visual large model Token includes: Upon receiving an image processing request, the master node inputs the image to be processed into a dynamic resolution predictor module to determine an optimal resolution of the image to be processed, and resizes the image to be processed based on the optimal resolution to obtain a resized image; the image processing request is sent by a user; the image processing request includes an image processing requirement and a queue of images to be processed including a plurality of images to be processed; When the optimal resolution is greater than the preset resolution value, the resized image is added to the waiting queue; The inference service planning device uses a dynamic programming algorithm to determine a dynamic planning result based on the waiting queue and image processing requirements; the dynamic planning result includes the token distribution of all requests in the waiting queue and the edge device corresponding to each token packet; The Token Heterogeneous Packaging Strategy Module packages the first request in the waiting queue into multiple token packages based on the resized image and the token distribution of the first request in the waiting queue. According to the dynamic planning results, all token packets are sent to different edge devices for processing; Obtain and integrate the calculation results of the large visual model in all multiple edge devices that receive the same label, and obtain the image classification result of the image to be processed corresponding to the first request in the waiting queue.

2. The image classification method based on visual large model Token parallelization according to claim 1 is characterized in that: The image processing requirements include a start time, a deadline, and a utility.

3. The image classification method based on visual large model Token parallelization according to claim 1 is characterized in that The image classification method based on the parallelization of visual large model Token also includes: When the optimal resolution is less than or equal to the preset value, the resized image is sent to any idle edge device, and the calculation result of the visual large model in the edge device is obtained as the image classification result of the image to be processed.

4. The image classification method based on visual large model Token parallelization according to claim 1 is characterized in that The dynamic resolution predictor module is a ResNet network; The dynamic resolution predictor module is: represents the dynamic resolution predictor module; C x represents the predicted probability of resolution; Represent the prediction probabilities of the 1st, 2nd, and mth resolutions respectively; In order to select the resolution with the highest probability, the one_hot function needs to be used; The one_hot function is: h=one_hot(argmax[g i +log C i ]); The loss function when training the dynamic resolution predictor module is: in, y i =f(X i ); Mr i =-log(-log) u~U(0,1); Where L is the loss function when training the dynamic resolution predictor module; is the cross loss entropy; L ron is the first loss; η is the hyperparameter; η is the first loss L ron Impact factor; is the predicted label; y true is the true label; y i represents the output result of inputting the m-th resolution image into the visual large model; f(X i ) represents the visual large model; X i represents the m-th resolution image; E(*) is the mean function; F is the average GFLOPs at m resolutions; β is the GFLOPs target value; G max is the model GFLOPs at maximum resolution; G min h is the GFLOPs of the model with the minimum resolution; i is the resolution image X i The corresponding predicted probability under G i GFLOPs of the model corresponding to each resolution; g i is Gaussian noise; τ is the temperature coefficient in the softmax function; exp(*) is the calculation formula of e^x; u is a uniformly distributed random value; U(0,1) is a uniform distribution; g i is Gaussian noise; C i The value predicted for dynamic resolution.

5. The image classification method based on visual large model Token parallelization according to claim 1 is characterized in that: The large visual model includes: a linear mapping layer, a transformer encoder and a classification head network connected in sequence; The workflow of the visual large model is as follows: Divide the input image into multiple image blocks and determine the position code of each image block; Input multiple image blocks into the linear mapping layer to obtain the token corresponding to each image block; After combining multiple tokens with the position codes of the corresponding image blocks, a sequence of image block embedding vectors is formed; Input the image block embedding vector sequence into the transformer encoder to obtain the image encoding; The image encoding is input to the classification head network to obtain the classification result of the input image.

6. The image classification method based on visual large model Token parallelization according to claim 5 is characterized in that: The transformer encoder includes a plurality of sequentially connected multi-head self-attention-multi-layer perception machine layers; Any of the multi-head self-attention-multi-layer perceptron layers comprises a multi-head self-attention layer and a multi-layer perceptron connected in sequence; The multi-head self-attention layer includes a normalization layer of residual connections and a multi-head self-attention mechanism; The multi-layer perceptron includes a normalization layer with residual connections and a multi-layer perceptron.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that the processor executes the computer program to implement the image classification method based on visual large model Token parallelization as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image classification method based on visual large model Token parallelization according to any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image classification method based on visual large model Token parallelization according to any one of claims 1 to 6 is implemented.

Images