Basic model-oriented scene adaptive edge cloud collaborative inference system and method

Abstract

The invention relates to a basic model-oriented scene self-adaptive edge cloud collaborative reasoning system and method. The system comprises a scene self-adaptive edge side model customization component and a self-adaptive reasoning component; the scene-adaptive side model customization assembly extracts a hybrid expert model from a basic model deployed at a cloud side through a super-class routing and model distillation method, and then selects a proper expert module to compress and finely adjust a side customization model by using a small amount of label-free original data on edge equipment. And the adaptive reasoning component processes the reasoning task in cooperation with the side customization model and the basic model deployed on the cloud side. And uploading a reasoning result of the side customized model to a decision module, calculating a confidence score, and determining whether a data block selector needs to be uploaded or not, if so, uploading local and important data blocks to a basic model deployed at a cloud side, so that the effects of reducing transmission overhead and protecting data privacy are achieved.

Description

Technical Field

[0001] The present invention relates to artificial intelligence and edge computing technology, and in particular to a scene-adaptive edge-cloud collaborative reasoning system and method for a basic model. Background Art

[0002] As an emerging artificial intelligence technology, the basic model has received widespread attention; the basic model can effectively process multiple data modalities, such as text, images, videos, audio, etc.; the basic model obtains rich knowledge reserves through pre-training on large-scale data sets, supports multiple downstream tasks, and greatly reduces the data requirements and training costs for specific tasks.

[0003] Based on the above characteristics, the basic model has some typical applications on edge devices; for example:

[0004] ① Intelligent monitoring, using smart cameras and other sensors to identify unusual activities or sounds;

[0005] ② Intelligent assistants use multimodal information such as data records, texts and pictures on mobile phones to analyze user portraits and provide intelligent services;

[0006] ③Virtual / augmented reality, allowing users in different regions to share and modify virtual / augmented reality content in real time.

[0007] Although the basic model has excellent performance, with the continuous improvement of people's production and living standards, the requirements for various intelligent technologies are also gradually increasing, which makes the deployment solutions on existing edge devices face many challenges.

[0008] Due to the limited computing power and storage resources of edge devices, the memory overhead of the basic model cannot be met, making it difficult for them to fully realize their potential in existing practical applications.

[0009] At present, most research uses technologies such as data compression, model lightweighting, and model partitioning to deploy basic models on edge devices; some work directly compresses the original data and uploads it to the cloud server for processing. Although it reduces the transmission cost, simple data compression will lead to the loss of important information and greatly affect the reasoning accuracy.

[0010] There are also some works that use model lightweight techniques (such as pruning, quantization, and knowledge distillation) to compress the basic model and perform reasoning on edge devices, but this approach will cause loss of accuracy and generalization of the basic model.

[0011] Some other works use cross-layer or hierarchical methods to divide the model and perform edge-cloud collaborative reasoning, but they are more suitable for models with convolutional neural network (CNN) architecture; the basic models mostly use the Transformer architecture, and its self-attention mechanism cannot be divided, making the cross-layer method inapplicable.

[0012] In addition, the basic model lacks dimension reduction operations and the input and output shapes remain consistent, resulting in poor optimization results for the hierarchical approach. Summary of the invention

[0013] In order to solve the above-mentioned technical problems, the present invention provides a scene-adaptive edge-cloud collaborative reasoning system and method for a basic model. By utilizing a single powerful basic model, a lightweight model is customized based on the data types and task requirements of different scenarios to perform edge-cloud collaborative reasoning. The system is suitable for a variety of applications such as intelligent monitoring, intelligent assistants, virtual / augmented reality, etc., and effectively solves the problems of high computing overhead and transmission delay caused by insufficient computing power of edge devices. At the same time, local data blocks of the original data are uploaded to protect data privacy.

[0014] To achieve the above object, the technical solution adopted by the present invention is: firstly, a scene-adaptive edge-cloud collaborative reasoning system for a basic model is provided, including: a scene-adaptive edge-side model customization component and an adaptive reasoning component;

[0015] The scene-adaptive side model customization component includes:

[0016] Hybrid expert model extraction module: used to extract the hybrid expert model from the basic model deployed on the cloud side through superclass routing and model distillation methods, and divide the knowledge of specific fields for different expert modules in the hybrid expert model in the form of superclasses. The superclass responsible for each expert module comes from the category of the public dataset. The hybrid expert model serves as a pre-trained model for the side customized model.

[0017] Edge customized model fine-tuning module: After completing the extraction of the hybrid expert model, it selects an expert module with the highest importance from each hybrid expert layer according to the data type and task requirements of the edge device, compresses the hybrid expert model, and uses the unlabeled raw data of the edge device to fine-tune the edge customized model. This fine-tuning method combines model distillation and contrastive learning, using the basic model deployed on the cloud side as the teacher model for model distillation. At the same time, the text feature library T on the cloud side is used to select the text features required for contrastive learning.

[0018] The adaptive reasoning component is used to coordinate the edge-side customized model and the basic model deployed on the cloud side to process reasoning tasks; including:

[0019] Upload decider: used to evaluate the confidence score of the edge-customized model inference result. When the confidence score is higher than the set confidence threshold, the edge-customized model inference result is adopted. Otherwise, the data sample is input into the upload data block selector.

[0020] Upload data block selector: used to evaluate the importance of data blocks of input data samples, select and upload local and important data sample blocks to the basic model deployed on the cloud side;

[0021] Collaborative strategy adjuster: used to estimate the real-time network bandwidth to update the transmission delay, query the threshold search table, calculate the predicted end-to-end inference delay, and dynamically adjust the confidence threshold of the upload decider and the number of data sample blocks of the upload data block selector according to different data types and task requirements to balance the delay and accuracy of the inference task.

[0022] Preferably, the hybrid expert module comprises several backbone network layers, a superclass routing, several hybrid expert layers and a feature mapping layer, and the hybrid expert layer comprises several expert modules responsible for different superclasses.

[0023] Preferably, the importance calculation process of the edge customized model fine-tuning module is as follows:

[0024] The importance is evaluated by calculating the probability of each expert module being selected using the data samples of the edge device. The data samples are sent to the expert module e by superclass routing. j The importance is represented by I j , the calculation formula is:

[0025] P i =softmax(W r ),

[0026]

[0027] Among them, P i = {P i,1 ,P i,2 ,...,P i,N , represents the probability that data sample i is assigned to N expert modules, W r is the N-dimensional probability distribution vector output by the superclass routing, P i,j The data sample i is sent to the expert module e by the superclass routing j The probability of , i is the data sample number, j is the expert module number.

[0028] Preferably, the fine-tuning process of the edge customized model fine-tuning module is as follows:

[0029] The unlabeled raw data of the edge device is used to fine-tune the edge customized model. This fine-tuning method combines model distillation and contrastive learning. The basic model deployed on the cloud side is used as the teacher model for model distillation. At the same time, the text feature library on the cloud side is used to select the text features required for contrastive learning.

[0030] First, the data samples are input into the basic model deployed on the cloud side and the customized model on the edge side, and the sample features extracted by the basic model are calculated. And sample features extracted by the side customized model The mean squared error loss (Mean Squared Error loss) is used as the loss function L for model distillation. dist , the calculation formula is:

[0031]

[0032] At the same time, in order to adapt to the unlabeled data environment of edge devices, the loss function L of contrastive learning is set con , query the sample features extracted from the text feature library and the basic model Most similar text features Closely define the sample features extracted by the side-customized model and text features The distance between the two, the loss function L of contrastive learning con The calculation formula is:

[0033] L con =L I,T +L T,I ,

[0034]

[0035] Among them, sim(·) is the cosine similarity of the feature, and τ is the temperature coefficient;

[0036] The loss function L for side-by-side custom model fine-tuning is the loss function L for model distillation dist And the loss function L of contrastive learning con Combination of:

[0037]

[0038] in, is the weight of each data sample.

[0039] Preferably, the confidence score calculation process of the upload decision module is as follows:

[0040] The data sample is input into the edge customized model, the sample features are output, the cosine similarity between the sample and each text feature in the text feature library is calculated, the entropy value is calculated based on the cosine similarity distribution, and the entropy value is normalized to obtain the confidence score of the input data sample.

[0041] Preferably, the process of evaluating the importance of the data blocks of the input data samples by the upload data block selector module is as follows:

[0042] The cosine similarity between the class token and the data tokens of the last layer of the side custom model is used as the category attention score A cls , represents the importance of the data block of the input data sample, and the calculation formula is:

[0043]

[0044] Among them, q cls Mark the class token for the category, K T is the key matrix, i.e. the set of all input data labels, and d is the dimension of category labels and data labels.

[0045] Preferably, the end-to-end reasoning delay of the collaborative strategy adjuster is calculated as follows:

[0046] Regularly collect raw data samples from edge devices, calculate the ratio of data samples processed by the edge-side custom model and the basic model on the cloud side r(thre), the overall accuracy acc(thre), and the edge-side processing delay l e , transmission delay tr (thre), cloud-side processing delay l c , calculate the end-to-end inference latency And store it in the threshold search table, the calculation formula is as follows:

[0047]

[0048] The present invention also provides a scene-adaptive edge-cloud collaborative reasoning method for a basic model, which is performed in the scene-adaptive edge-cloud collaborative reasoning system for a basic model, and includes the following steps:

[0049] S1 extracts a hybrid expert model from the basic model deployed on the cloud side through superclass routing and model distillation, and divides the knowledge in specific fields for different expert modules in the hybrid expert model in the form of superclasses. The superclass that each expert module is responsible for comes from the category of the public dataset. The hybrid expert model serves as a pre-trained model for the edge customized model.

[0050] After S2 completes the extraction of the hybrid expert model, it selects an expert module with the highest importance from each hybrid expert layer according to the data type and task requirements of the edge device, compresses the hybrid expert model, and uses the unlabeled raw data of the edge device to fine-tune the edge-side customized model. This fine-tuning method combines model distillation and contrastive learning, using the basic model deployed on the cloud side as the teacher model for model distillation. At the same time, the text feature library T on the cloud side is used to select the text features required for contrastive learning.

[0051] S3: The edge-customized model divides the input data sample into multiple data blocks and calculates the preliminary reasoning results; then, the preliminary reasoning results are transmitted to the upload decision maker;

[0052] S4 evaluates the confidence score of the edge customized model inference result. When the confidence score is higher than the set confidence threshold, the edge customized model inference result is adopted. Otherwise, the data sample is input into the upload data block selector.

[0053] S5 evaluates the importance of the data blocks of the input data samples, selects and uploads local and important data sample data blocks to the basic model deployed on the cloud side, and uses the basic model deployed on the cloud side to infer the results;

[0054] S6 estimates the real-time network bandwidth to update the transmission delay, queries the threshold search table, calculates the predicted end-to-end inference delay, and dynamically adjusts the confidence threshold of the upload decision maker and the number of data sample data blocks of the upload data block selector according to different data types and task requirements to balance the delay and accuracy of the inference task.

[0055] The above technical solution has the following advantages or beneficial effects:

[0056] The present invention designs a scene-adaptive edge-cloud collaborative reasoning system and method for basic models, which realizes efficient and accurate edge-cloud collaborative reasoning of basic models on edge devices in different scenes, and has the following innovations:

[0057] ① The present invention solves the deployment problem of the basic model on edge devices and achieves efficient and accurate reasoning.

[0058] ② The present invention realizes the use of a single basic model to adaptively customize the edge lightweight model based on the data types and task requirements of different scenarios.

[0059] ③The present invention realizes the importance assessment of the original data on the edge, and only uploads local and important data blocks, reducing transmission delay and protecting data privacy.

[0060] ④ The present invention realizes dynamic adjustment of collaborative reasoning strategy to balance the requirements of reasoning latency and accuracy.

[0061] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present invention and should not be regarded as limiting the scope of the present invention.

[0063] Figure 1 The overall architecture diagram of a scene-adaptive edge-cloud collaborative reasoning system for a basic model provided by the present invention;

[0064] Figure 2 A diagram of the architecture of the hybrid expert model provided by the present invention;

[0065] Figure 3 A flow chart of fine-tuning the side customization model provided by the present invention; DETAILED DESCRIPTION

[0066] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0067] The present invention first provides a scene-adaptive edge-cloud collaborative reasoning system for basic models. The overall architecture of the system is as follows: Figure 1 As shown, it includes: a scene-adaptive side model customization component and an adaptive reasoning component.

[0068] The scenario-adaptive edge model customization component utilizes a single powerful basic model and a small amount of unlabeled edge device raw data to output edge customized models for edge devices with different data types and task requirements, including a hybrid expert model extraction module and an edge customized model fine-tuning module.

[0069] Existing methods can obtain lightweight edge-side customized models through model compression and fine-tuning technology, but this method has high training cost and requires a large amount of training data. Therefore, the present invention extracts a hybrid expert model from the basic model deployed on the cloud side, compresses it as a pre-trained model, and fine-tunes a lightweight edge-side customized model.

[0070] The architecture of the hybrid expert model extraction module is as follows Figure 2As shown, it includes several backbone network layers, a superclass routing, several hybrid expert layers and a feature mapping layer; the hybrid expert layer includes several expert modules, and different expert modules are activated according to different data types and task requirements, and each expert module has professional knowledge in a specific field. The backbone network layer receives data samples as input and extracts low-level sample features. The superclass routing selects the most suitable expert module according to the low-level sample features. The expert module extracts high-level sample features. The feature mapping layer maps the high-level sample features to the same number of dimensions as the basic model as the final output sample features.

[0071] This module is used for training through superclass routing and model distillation methods. Mixed Expert Model , group the categories of the public dataset into superclasses, freeze the backbone network layer, and use supervised learning methods to train superclass routing and expert modules based on the basic model to ensure that each expert module handles a specific superclass and retains the performance of the basic model to the greatest extent. For datasets that do not have superclass classification, such as the image dataset ImageNet, use the basic model to calculate the confusion matrix on the dataset validation set, and construct a confusion graph from it. Apply the graph clustering algorithm to obtain the superclass grouping of the dataset.

[0072] The present invention extracts a hybrid expert model from a large-scale public dataset, and selects an image dataset ImageNet (1000 categories, 10 superclasses) and an audio dataset Audioset (1000 categories, 12 superclasses) to implement two types of tasks: image and audio recognition.

[0073] The present invention selects CLIP (Contrastive Language-Image Pre-Training), a basic model multimodal pre-trained neural network, and ImageBind, a method for learning joint feature embedding, to extract a hybrid expert model. Among them, CLIP has an image encoder and a text encoder, and can perform image recognition tasks; ImageBind has an image encoder, an audio encoder and a text encoder, and can perform image and audio recognition tasks. Specifically, taking the image recognition task as an example, the basic model uses a text encoder to calculate text features of a series of categories, and uses an image encoder to calculate image features of image data. Finally, the text feature with the highest cosine similarity with the image feature is calculated and selected, and the category corresponding to the text feature is used as the recognized image category.

[0074] The extracted hybrid expert model has 12 Transformer layers, and the last 3 layers are hybrid expert layers. The hidden embedding dimension in the Transformer layer is 192, the number of heads of the multi-head self-attention mechanism is 3, and the dimension of the multi-layer perceptron is 4 times the hidden embedding dimension. For the image recognition task, 10 expert modules are set for each hybrid expert layer, the input image size is 224×224×3, and the data block size is 16×16; for the audio recognition task, 12 expert modules are set for each hybrid expert layer, the input data size is 128×204×3, and the data block size is 16×16.

[0075] The edge customized model fine-tuning module is used to complete the extraction of the hybrid expert model, select an expert module with the highest importance from each hybrid expert layer according to the data type and task requirements of the edge device, compress the hybrid expert model, and use the unlabeled original data of the edge device to fine-tune the edge customized model; at the same time, the size of the edge customized model can be adjusted according to the scenario and computing power of the edge device.

[0076] The importance calculation process is as follows:

[0077] The importance is evaluated by calculating the probability of each expert module being selected using the data samples of the edge device. The data samples are sent to the expert module e by superclass routing. j The importance is represented by I j , the calculation formula is:

[0078] P i =softmax(W r ),

[0079]

[0080] Among them, P i = {P i,1 ,P i,2 ,...,P i,N}, represents the probability that data sample i is assigned to N expert modules, W r is the N-dimensional probability distribution vector output by the superclass routing, P i,j The data sample i is sent to the expert module e by the superclass routing j The probability of , i is the data sample number, j is the expert module number.

[0081] The present invention collects multiple categories from multiple edge devices and public data sets, and the text encoder of the basic model calculates the text features of these categories to form a text feature library T. At the same time, it supports the free addition of multiple new categories. During the reasoning process, the edge customized model extracts image or audio features, and image or audio recognition is achieved by selecting the text feature with the highest cosine similarity with the image or audio feature from the text feature library T.

[0082] like Figure 3 As shown, the present invention uses the unlabeled raw data of the edge device to fine-tune the edge customized model. The fine-tuning method combines model distillation and contrastive learning. The basic model deployed on the cloud side is used as the teacher model for model distillation. At the same time, the text feature library T on the cloud side is used to select the text features required for contrastive learning.

[0083] The fine-tuning process is as follows: First, the data samples are input into the basic model and the side customized model respectively, and the sample features extracted by the basic model are calculated. And sample features extracted by the side customized model The mean squared error loss (MeanSquared Error loss) is used as the loss function L for model distillation. dist , the calculation formula is:

[0084]

[0085] At the same time, in order to effectively utilize the unlabeled raw data of edge devices, the loss function L of contrastive learning is set con , query the sample features extracted from the text feature library T and the basic model Most similar text features Closely follow the sample features extracted by the side customized model and text features The distance between the two, the loss function L of contrastive learning con The calculation formula is:

[0086] L con =L I,T +L T,I ,

[0087]

[0088] Among them, sim(·) is the cosine similarity of the feature, and τ is the temperature coefficient;

[0089] The loss function L for side-by-side custom model fine-tuning is the loss function L for model distillation dist And the loss function L of contrastive learning con Combination of:

[0090]

[0091] in, is the weight of each data sample.

[0092] The adaptive reasoning component dynamically balances the latency and accuracy of reasoning tasks by flexibly adjusting the collaborative reasoning strategy of the edge-side customized model and the basic model deployed on the cloud side. It mainly consists of three modules: upload decider, upload data block selector, and collaborative strategy adjuster:

[0093] Existing methods use the output results of traditional models for evaluation, which is not applicable to the basic model deployed on the cloud side. This is because the traditional model takes data samples as input and outputs a one-hot encoding, that is, an N-bit feature vector is used to encode N states, where each state represents the confidence score of each category in the recognition task. However, the basic model deployed on the cloud side calculates data samples as high-dimensional sample features through image or audio encoders, and it is impossible to directly calculate the confidence score of the inference result from them.

[0094] Upload decider: used to evaluate the confidence score of the edge-customized model inference result. First, the data sample is input into the edge-customized model, the image features or audio features are calculated, and the cosine similarity between them and each text feature in the text feature library T is calculated. The entropy value is calculated based on the cosine similarity distribution, and the entropy value is normalized to obtain the confidence score of the input data sample. When the confidence score is higher than the set confidence threshold, the edge-customized model inference result is used. Otherwise, the data sample is input into the upload data block selector.

[0095] Upload data block selector: Evaluate the importance of data blocks of input data samples, select and upload data blocks of local and important data samples to the basic model deployed on the cloud side to reduce transmission costs and protect the data privacy of edge devices. Specifically, when performing inference, the edge-side customized model will divide the input data samples into multiple data blocks. Taking the image recognition task as an example, the edge-side customized model will divide the image data samples of 224×224 size into multiple 16×16 size data blocks (patches). In the inference stage, each data block will be mapped to a high-dimensional data tag (tokens). These tokens are input into the model together with a specially introduced class tag (class token) for calculation. The class tag is used to aggregate the overall information of the data sample and output the final sample features. By calculating the cosine similarity between each data block and the class tag, the similarity between the class tag and the data block can be quantified, thereby determining which data blocks contribute more to the final result.

[0096] The process of evaluating the importance of a data block of an input data sample is as follows:

[0097] The cosine similarity between the class token and the data tokens of the last layer of the side custom model is used as the category attention score A cls , represents the importance of the data block of the input data sample, and the calculation formula is:

[0098]

[0099] Among them, q cls Mark the class token for the category, K T is the key matrix, i.e. the set of all input data labels, and d is the dimension of category labels and data labels.

[0100] Collaborative strategy adjuster: In order to adapt to different edge device scenarios and network transmission conditions, the present invention designs a collaborative strategy adjuster. When performing reasoning, the present invention updates the transmission delay by estimating the real-time network bandwidth, queries the threshold search table, and calculates the predicted end-to-end reasoning delay. And according to the task requirements of different edge devices, including transmission delay constraints, accuracy constraints, etc., the confidence threshold of the upload decision maker and the number of data sample data blocks of the upload data block selector are dynamically adjusted to balance the transmission delay and reasoning accuracy, and realize efficient and accurate edge-cloud collaborative reasoning.

[0101] The end-to-end inference latency is calculated as follows:

[0102] Regularly collect raw data samples from edge devices, calculate the ratio of data samples processed by the edge-side custom model and the basic model on the cloud side r(thre), the overall accuracy acc(thre), and the edge-side processing delay l e , transmission delay tr (thre), cloud-side processing delay l c , calculate the end-to-end inference latency And store it in the threshold search table, the calculation formula is as follows:

[0103]

[0104] The present invention also provides a scene-adaptive edge-cloud collaborative reasoning method for a basic model, which is performed in the scene-adaptive edge-cloud collaborative reasoning system for a basic model, and includes the following steps:

[0105] S1 extracts a hybrid expert model from the basic model deployed on the cloud side through superclass routing and model distillation, and divides the knowledge in specific fields for different expert modules in the hybrid expert model in the form of superclasses. The superclass that each expert module is responsible for comes from the category of the public dataset. The hybrid expert model serves as a pre-trained model for the edge customized model.

[0106] After S2 completes the extraction of the hybrid expert model, it selects an expert module with the highest importance from each hybrid expert layer according to the data type and task requirements of the edge device, compresses the hybrid expert model, and uses the unlabeled raw data of the edge device to fine-tune the edge-side customized model. This fine-tuning method combines model distillation and contrastive learning, using the basic model deployed on the cloud side as the teacher model for model distillation. At the same time, the text feature library on the cloud side is used to select the text features required for contrastive learning.

[0107] S3: The edge-customized model divides the input data sample into multiple data blocks and calculates the preliminary reasoning results; then, the preliminary reasoning results are transmitted to the upload decision maker;

[0108] S4 evaluates the confidence score of the edge customized model inference result. When the confidence score is higher than the set confidence threshold, the edge customized model inference result is adopted. Otherwise, the data sample is input into the upload data block selector.

[0109] S5 evaluates the importance of the data blocks of the input data samples, selects and uploads local and important data sample data blocks to the basic model deployed on the cloud side, and uses the basic model deployed on the cloud side to infer the results;

[0110] S6 estimates the real-time network bandwidth to update the transmission delay, queries the threshold search table, calculates the predicted end-to-end inference delay, and dynamically adjusts the confidence threshold of the upload decision maker and the number of data sample data blocks of the upload data block selector according to different data types and task requirements to balance the delay and accuracy of the inference task.

[0111] Experimental results show that the present invention significantly reduces latency in image and audio recognition tasks of different data types and task requirements, while maintaining accuracy comparable to that of the basic model. Specifically, in terms of latency performance, the present invention is 2.19 times faster than cloud-side reasoning on average, 8.75 times faster than edge-side reasoning on average, and 4.47 times faster than edge-cloud collaborative reasoning methods based on early exit on average. Similarly, the present invention also significantly reduces latency under different network transmission conditions. Under low (5Mbps), medium (25Mbps), and high (55Mbps) network bandwidth conditions, the present invention achieves the lowest latency compared to cloud-side reasoning, edge-side reasoning, and traditional edge-cloud collaborative reasoning methods. Under low bandwidth conditions, the latency of the present invention is reduced by 1.85 times, 4.82 times, and 2.07 times, respectively, compared to other methods. In terms of accuracy, the present method is 4.0% higher than edge-side reasoning on average, and 3.1% higher than edge-cloud collaborative reasoning methods based on early exit on average.

[0112] The above technical solution has the following advantages or beneficial effects: Compared with the prior art, the present invention is oriented to the basic model of the Transformer architecture, outputs a scene-adaptive edge-customized model, and thus realizes accurate and efficient edge-cloud collaborative reasoning.

[0113] The present invention designs a scene-adaptive edge-cloud collaborative reasoning system for basic models to solve the current deployment problem of basic models on edge devices.

[0114] The present invention utilizes a single powerful basic model to adaptively output edge-customized models based on the data types and task requirements in different scenarios; uploads local and important data blocks of the original data to reduce transmission costs and protect data privacy; and adaptively adjusts the edge-cloud collaborative reasoning strategy to achieve efficient and accurate reasoning.

[0115] At the same time, the present invention supports further expansion and application of the basic model in applications such as intelligent monitoring, intelligent assistants, virtual / augmented reality, etc.

[0116] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A scene-adaptive edge-cloud collaborative reasoning system for a basic model, comprising: Scene-adaptive side model customization components and adaptive reasoning components; The scene-adaptive side model customization component includes: Hybrid expert model extraction module: extracts the hybrid expert model from the basic model deployed on the cloud side through superclass routing and model distillation methods, and divides the knowledge in specific fields for different expert modules in the hybrid expert model in the form of superclasses. The superclass that each expert module is responsible for comes from the category of the public dataset. The hybrid expert model serves as a pre-trained model for the side customized model. Edge-side customized model fine-tuning module: After completing the extraction of the hybrid expert model, it selects an expert module with the highest importance from each hybrid expert layer according to the data type and task requirements of the edge device, compresses the hybrid expert model, and uses the unlabeled raw data of the edge device to fine-tune the edge-side customized model. This fine-tuning method combines model distillation and contrastive learning, using the basic model deployed on the cloud side as the teacher model for model distillation. At the same time, the text feature library T on the cloud side is used to select the text features required for contrastive learning. The adaptive reasoning component includes: Upload decider: used to evaluate the confidence score of the edge-customized model inference result. When the confidence score is higher than the set confidence threshold, the edge-customized model inference result is adopted. Otherwise, the data sample is input into the upload data block selector. Upload data block selector: used to evaluate the importance of data blocks of input data samples, select and upload local and important data sample blocks to the basic model deployed on the cloud side; Collaborative strategy adjuster: used to estimate the real-time network bandwidth to update the transmission delay, query the threshold search table, calculate the predicted end-to-end inference delay, and dynamically adjust the confidence threshold of the upload decider and the number of data sample blocks of the upload data block selector according to different data types and task requirements to balance the delay and accuracy of the inference task.

2. The scene-adaptive edge-cloud collaborative reasoning system for basic models as described in claim 1, characterized in that: The hybrid expert module includes several backbone network layers, a superclass routing, several hybrid expert layers and a feature mapping layer, and the hybrid expert layer includes several expert modules responsible for different superclasses.

3. The scene-adaptive edge-cloud collaborative reasoning system for basic models as described in claim 1, characterized in that: The importance calculation process described in the side-customized model fine-tuning module is as follows: The importance is evaluated by calculating the probability of each expert module being selected using the data samples of the edge device. The data samples are sent to the expert module e by superclass routing. j The importance is represented by I j , the calculation formula is: P i =softmax(W r ), Among them, P i = {P i,1 ,P i,2 ,...,P i,N }, represents the probability that data sample i is assigned to N expert modules, W r is the N-dimensional probability distribution vector output by the superclass routing, P i,j The data sample i is sent to the expert module e by the superclass routing j The probability of , i is the data sample number, j is the expert module number.

4. The scene-adaptive edge-cloud collaborative reasoning system for basic models as described in claim 1, characterized in that: The fine-tuning process of the side-customized model fine-tuning module is as follows: Input the data samples into the basic model and the side customized model respectively, and calculate the sample features extracted by the basic model And sample features extracted by the side customized model The mean squared error loss (MSE) is used as the loss function L for model distillation. dist , the calculation formula is: At the same time, in order to adapt to the unlabeled data environment of edge devices, the loss function L of contrastive learning is set con , query the sample features extracted from the text feature library and the basic model Most similar text features Closely define the sample features extracted by the side-customized model and text features The distance between the two, the loss function L of contrastive learning con The calculation formula is: L con =L I,T +L T,I , Among them, sim(·) is the cosine similarity of the feature, and τ is the temperature coefficient; The loss function L for side-by-side custom model fine-tuning is the loss function L for model distillation dist And the loss function L of contrastive learning con Combination of: in, is the weight of each data sample.

5. The scene-adaptive edge-cloud collaborative reasoning system for basic models as described in claim 1, characterized in that: The confidence score calculation process of the upload decision maker module is as follows: The data sample is input into the edge customized model, the sample features are output, the cosine similarity between the sample and each text feature in the text feature library is calculated, the entropy value is calculated based on the cosine similarity distribution, and the entropy value is normalized to obtain the confidence score of the input data sample.

6. The scene-adaptive edge-cloud collaborative reasoning system for basic models as described in claim 1, characterized in that: The upload block selector module evaluates the importance of the data blocks of the input data sample as follows: The cosine similarity between the class token and the data tokens of the last layer of the side custom model is used as the category attention score A cls , represents the importance of the data block of the input data sample, and the calculation formula is: Among them, q cls Mark the class token for the category, K T is the key matrix, i.e. the set of all input data labels, and d is the dimension of category labels and data labels.

7. The scene-adaptive edge-cloud collaborative reasoning system for basic models according to claim 1, characterized in that: The end-to-end reasoning delay of the collaborative strategy adjuster is calculated as follows: Regularly collect raw data samples from edge devices, calculate the ratio of data samples processed by the edge-side custom model and the basic model on the cloud side, r(thre), the overall accuracy acc(thre), and the edge-side processing delay l e , transmission delay tr (thre), cloud-side processing delay l c , calculate the end-to-end inference latency And store it in the threshold search table, the calculation formula is as follows:

8. A scene-adaptive edge-cloud collaborative reasoning method for a basic model, performed in the scene-adaptive edge-cloud collaborative reasoning system for a basic model, comprising the following steps: S1 extracts a hybrid expert model from the basic model deployed on the cloud side through superclass routing and model distillation, and divides the knowledge in specific fields for different expert modules in the hybrid expert model in the form of superclasses. The superclass that each expert module is responsible for comes from the category of the public dataset. The hybrid expert model serves as a pre-trained model for the edge customized model. After S2 completes the extraction of the hybrid expert model, it selects an expert module with the highest importance from each hybrid expert layer according to the data type and task requirements of the edge device, compresses the hybrid expert model, and uses the unlabeled raw data of the edge device to fine-tune the edge-side customized model. This fine-tuning method combines model distillation and contrastive learning, using the basic model deployed on the cloud side as the teacher model for model distillation. At the same time, the text feature library on the cloud side is used to select the text features required for contrastive learning. S3: The edge customized model divides the input data sample into multiple data blocks and calculates the preliminary reasoning results; then, the preliminary reasoning results are transmitted to the upload decision maker; S4 evaluates the confidence score of the edge customized model inference result. When the confidence score is higher than the set confidence threshold, the edge customized model inference result is adopted. Otherwise, the data sample is input into the upload data block selector. S5 evaluates the importance of the data blocks of the input data samples, selects and uploads local and important data sample blocks to the basic model deployed on the cloud side, and uses the basic model deployed on the cloud side to infer the results; S6 estimates the real-time network bandwidth to update the transmission delay, queries the threshold search table, calculates the predicted end-to-end inference delay, and dynamically adjusts the confidence threshold of the upload decision maker and the number of data sample data blocks of the upload data block selector according to different data types and task requirements to balance the delay and accuracy of the inference task.

Images