Method for executing edge cloud fusion visual cue large language model
By training large language model feature encoder and small feature extraction models in edge devices, and combining visual cues image encoder, fully connected linear projector and large language model database, the visual cues large language model with edge cloud fusion is realized, solving the problem of increased computing resource requirements and delays of large language models when processing visual data, improving performance and user experience.
Patent Information
- Application Number
- CN202410559717.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-01-11
- Filing Date
- 2024-05-08
- Publication Date
- 2025-03-11
AI Technical Summary
Large language models face problems such as increased computing resource requirements and delays when processing visual data, which affects user experience, especially in applications with high immediate response requirements.
By training large language model feature encoder and small feature extraction models in edge devices, and combining visual cues image encoder, fully connected linear projector and large language model database, a large visual cues large language model with edge cloud fusion is realized. This method offloads visual data processing to edge devices, reduces cloud computing load, and accelerates response through small feature extraction models.
Efficiently utilize visual data, reduce computing and latency problems, improve the performance and user experience of large language models, and provide solutions that instantly adapt to changes in network environments and system loads.
Smart Images
Figure CN119597951A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a large language model, and more particularly to a method for performing edge-cloud fusion of visually prompted large language models. Background Art
[0002] With the development of artificial intelligence, large language models (LLMs) have begun to play a key role in many applications such as information retrieval, sentiment analysis, and natural language understanding. However, these models usually focus on text data, while a large amount of visual data is basically untapped. Including visual cues in large language models can significantly improve the quality of predictions and provide more nuanced explanations for real-world scenarios.
[0003] However, incorporating visual data processing into large language models brings a new set of challenges, especially in terms of computational resource requirements and latency. Processing visual data requires a lot of computing power and may lead to increased latency, which is detrimental to user experience, especially in real-time applications where instant response is critical. Given these challenges, a new approach is needed to effectively utilize visual data in large language models while addressing both computational and latency issues. Summary of the invention
[0004] A method for executing a large language model of visual cues for edge-cloud fusion includes training a large language model feature encoder and a small feature extraction model in an edge device, inputting knowledge-based text cues into the large language model feature encoder in the edge device to generate multiple knowledge-based text embeddings, establishing a large language model database in the edge device based on the knowledge-based text embeddings, inputting text cues into the large language model feature encoder to generate a text query embedding, comparing the text query embedding with the large language model database to generate a first similarity score, and if the first similarity score is greater than a first threshold, inputting the text query embedding into the small feature extraction model to generate a first answer.
[0005] A method for performing edge-cloud fused visual cue large language model includes training a visual cue image encoder, a fully connected linear projector and a small feature extraction model in an edge device, inputting multiple knowledge-based image cues into the visual cue image encoder in the edge device to generate multiple knowledge-based image embeddings, establishing a large language model database in the edge device based on the knowledge-based image embeddings, inputting an image cue into the visual cue image encoder to generate a visual representation, inputting the visual representation into the fully connected linear projector in the edge device to generate an image query embedding, comparing the image query embedding with the large language model database to generate a second similarity score, and if the second similarity score is greater than a second threshold, inputting the image query embedding into the small feature extraction model to generate a second answer.
[0006] A method for executing a large language model of visual cues for edge-cloud fusion, comprising training a visual cue image encoder, a fully connected linear projector, a large language model feature encoder and a small feature extraction model in an edge device, inputting a plurality of knowledge-based image cues and a plurality of knowledge-based text cues into the visual cue image encoder and the large language model feature encoder in the edge device respectively to generate a plurality of knowledge-based image embeddings and a plurality of knowledge-based text embeddings, connecting the knowledge-based image embeddings and the knowledge-based text embeddings to generate a plurality of knowledge-based connected embeddings, and establishing a large language model in the edge device according to the knowledge-based connected embeddings. database, inputting an image prompt into the visual prompt image encoder in the edge device to generate a visual representation, inputting the visual representation into the fully connected linear projector in the edge device to generate an image query embedding, inputting a text prompt into the large language model feature encoder in the edge device to generate a text query embedding, concatenating the image query embedding and the text query embedding to generate a concatenated query embedding, comparing the concatenated query embedding with the large language model database to generate a third similarity score, and if the third similarity score is greater than a third threshold, inputting the concatenated query embedding into the small feature extraction model to generate a third answer. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 is a structural diagram of a large language model of visual cues integrated with edge and cloud according to an embodiment of the present invention.
[0008] Figure 2 It is a flowchart of a method for operating a large language model of visual prompts integrated with edge and cloud according to an embodiment of the present invention.
[0009] Figure 3is a flowchart of a method for running an edge-cloud fused visual cue large language model according to another embodiment of the present invention.
[0010] Figure 4 is a flowchart of a method for running an edge-cloud fused visual cue large language model according to another embodiment of the present invention.
[0011]
Explanation of symbols
[0012] 100: Large Language Models Fusion of Visual Cues on Edge and Cloud
[0013] 102: Visual Transformer
[0014] 104: Visual Abstraction Module
[0015] 106: Visual Cue Image Encoder
[0016] 108:Fully connected linear projector
[0017] 110: Small feature extraction model
[0018] 112: Large Feature Extraction Model
[0019] 114: Large Language Model Feature Encoder
[0020] 116: Large Language Model Database
[0021] 200,300,400: Method
[0022] S201, S202, S204, S206, S208, S210, S301, S302, S304, S306, S308, S310, S312, S401, S402, S403, S404, S405, S406, S407, S408, S409, S410, S412, S413, S414: Steps DETAILED DESCRIPTION
[0023] Edge devices are used to perform processing tasks, such as extracting features from visual cues, which reduces the computing load on the cloud and speeds up response time. In addition, offloading the computationally intensive large language model processing to the cloud ensures that edge devices will not fail to run, and users can enjoy a smooth experience. At the same time, the small feature extraction model and large language model database proposed in the present invention can also speed up the user's response time.
[0024] Furthermore, the proposed model demonstrates impressive flexibility by providing a solution that can instantly adapt to changes in network environment and system load. In an era where edge devices and network conditions vary greatly, this degree of adaptability is critical and ensures robust performance of language models in different scenarios. Therefore, the motivation of the present invention is to improve the capabilities of large language models by integrating visual data processing, and to achieve this integration in an efficient and flexible manner using edge-cloud fusion methods. The proposed edge-cloud fusion-perceived visual cue-based large language model can not only protect the privacy of customers when running large language models in edge devices, but also amortize the work in the cloud by uploading only useful embeddings.
[0025] Figure 1 1 is a structural diagram of a visual cue large language model 100 fused at the edge and cloud according to an embodiment of the present invention. The visual cue large language model 100 fused at the edge and cloud includes a visual cue image encoder 106, a fully connected linear projector 108 connected to the visual cue image encoder 106, a large language model (LLM) feature encoder 114, a large language model database 116, a small feature extraction model 110 connected to the fully connected linear projector 108 and the large language model feature encoder 114, and a large feature extraction model 112 connected to the fully connected linear projector 108 and the large language model feature encoder 114. The visual cue image encoder 106 may include a visual transformer 102 and a visual abstraction module 104. The visual cue image encoder 106, the fully connected linear projector 108, the large language model feature encoder 114, the large language model database 116, and the small feature extraction model 110 are trained in the cloud and executed in the edge device. The large feature extraction model 112 is trained and executed in the cloud.
[0026] Figure 2 2 is a flow chart of a method 200 for operating a large language model 100 for visual cues integrated with edge and cloud according to an embodiment of the present invention. The method 200 starts at step S201 and includes the following steps:
[0027] Step S201: training the large language model feature encoder 114 and the small feature extraction model 110;
[0028] Step S202: inputting the knowledge-based text prompt into the large language model feature encoder 114 to generate a plurality of knowledge-based text embeddings;
[0029] Step S204: Building a large language model database 116 in the edge device based on the knowledge-based text embedding;
[0030] Step S206: Inputting a text prompt to the large language model feature encoder 114 in the edge device to generate a text query embedding;
[0031] Step S208: Compare the text query embedding with the large language model database 116 in the edge device to generate a first similarity score; if the first similarity score is greater than a first threshold, proceed to step S210; and
[0032] Step S210: Input the text query embedding into the small feature extraction model 110 to generate a first answer.
[0033] In step S201, the large language model feature encoder 114 and the small feature extraction model 110 can be trained in the cloud. In step S202, the knowledge-based text prompt is input to the large language model feature encoder 114 to generate multiple knowledge-based text embeddings. In step S204, a large language model database 116 is established in the edge device based on these knowledge-based text embeddings. In step S206, the text prompt is input to the large language model feature encoder 114 in the edge device to generate a text query embedding. In step S208, the text query embedding is compared with the large language model database 116 in the edge device to generate a first similarity score. If the first similarity score is greater than the first threshold, then enter step S210. In step S210, the text query embedding is input to the small feature extraction model 110 to generate a first answer. The first answer is a quick response to the text query embedding and is only implemented by the large language model feature encoder 114 of the edge device.
[0034] Figure 3 3 is a flow chart of a method 300 for running an edge-cloud fused visual cue large language model 100 according to another embodiment of the present invention. The method 300 starts at step S301 and includes the following steps:
[0035] Step S301: training the visual cue image encoder 106, the fully connected linear projector 108 and the small feature extraction model 110;
[0036] Step S302: inputting the knowledge-based image cue into the visual cue image encoder 106 to generate a plurality of knowledge-based image embeddings;
[0037] Step S304: Building a large language model database 116 in the edge device based on the knowledge-based image embedding;
[0038] Step S306: Input the image cue to the visual cue image encoder 106 to generate a visual representation;
[0039] Step S308: input the visual representation to the fully connected linear projector 108 to generate an image query embedding;
[0040] Step S310: compare the image query embedding with the large language model database 116 in the edge device to generate a second similarity score; if the second similarity score is greater than a second threshold, proceed to step S312; and
[0041] Step S312: Input the image query embedding into the small feature extraction model 110 to generate a second answer.
[0042] In step S301, the visual cue image encoder 106, the fully connected linear projector 108, and the small feature extraction model 110 can be trained in the cloud. In step S302, the knowledge-based image cue is input to the visual cue image encoder 106 to generate multiple knowledge-based image embeddings. In step S304, a large language model database 116 is established in the edge device based on these knowledge-based image embeddings. In step S306, the image cue is input to the visual cue image encoder 106 to generate a visual representation. In step S308, the visual representation is input to the fully connected linear projector 108 to generate an image query embedding. In step S310, the image query embedding is compared with the large language model database 116 in the edge device to generate a second similarity score. If the second similarity score is greater than the second threshold, then enter step S312. In step S312, the image query embedding is input to the small feature extraction model 110 to generate a second answer. The second answer is a fast response to image query embedding and is only achieved by the large language model feature encoder 114 in the edge device.
[0043] Figure 4 4 is a flow chart of a method 400 for running an edge-cloud fused visual cue large language model 100 according to another embodiment of the present invention. The method 400 starts at step S401 and includes the following steps:
[0044] Step S401: training the large language model feature encoder 114, the visual cue image encoder 106, the fully connected linear projector 108 and the small feature extraction model 110;
[0045] Step S402: Establishing a large language model database 116;
[0046] Step S403: inputting a text prompt to the large language model feature encoder 114 in the edge device to generate a text query embedding;
[0047] Step S404: compare the text query embedding with the items in the large language model database 116 in the edge device to generate a first similarity score; if the first similarity score is greater than a first threshold, proceed to step S405; if the first similarity score is less than the first threshold, proceed to step S410;
[0048] Step S405: embedding the text query into the small feature extraction model 110 to generate a first answer;
[0049] Step S406: Input the image cue to the visual cue image encoder 106 in the edge device to generate a visual representation;
[0050] Step S407: input the visual representation into the fully connected linear projector 108 in the edge device to generate an image query embedding;
[0051] Step S408: Compare the image query embedding with the items in the large language model database 116 in the edge device to generate a second similarity score; if the second similarity score is greater than a second threshold, proceed to step S409; if the second similarity score is less than the second threshold, proceed to step S410;
[0052] Step S409: embedding the image query into the small feature extraction model 110 to generate a second answer;
[0053] Step S410: Connecting the text query embedding and the image query embedding to generate a connected query embedding;
[0054] Step S412: compare the connected query embedding with the items of the large language model database 116 in the edge device to generate a third similarity score; if the third similarity score is greater than a third threshold, proceed to step S413; if the third similarity score is less than the third threshold, proceed to step S414;
[0055] Step S413: Input the concatenated query embedding into the small feature extraction model 110 to generate a third answer; and
[0056] Step S414: embed the connected query into the large feature extraction model 112 in the cloud to return the fourth answer.
[0057] Before the actual measurement, the visual cue image encoder 106, the fully connected linear projector 108, the large language model feature encoder 114, the small feature extraction model 110 and the large feature extraction model 112 can be trained in the cloud in step S401. Then, the knowledge-based text prompts and the knowledge-based image prompts are input into the large language model feature encoder 114 in the edge device to generate multiple knowledge-based text embeddings and knowledge-based image embeddings. These knowledge-based image embeddings and these knowledge-based text embeddings are connected to generate multiple connected knowledge-based embeddings. In step S402, a large language model database 116 is constructed based on these connected knowledge-based embeddings, these knowledge-based image embeddings and these knowledge-based text embeddings.
[0058] During the actual measurement process, in step S403, the text prompt is input into the large language model feature encoder 114 in the edge device to generate a text query embedding. In step S404, the text query embedding is compared with the knowledge-based text embedding of the first library in the large language model database 116 to find the top k most similar embeddings in the first library. Then, in step S404, the embedding that best matches the text query embedding is selected from the top k most similar embeddings and compared with the text query embedding to generate a first similarity score. If the first similarity score is greater than the first threshold, proceed to step S405. If the first similarity score is less than the first threshold, proceed to step S410. In step S405, the text query embedding is input into the small feature extraction model 110 to generate a first answer. The first answer is a quick response to the text query embedding and is only implemented by the large language model feature encoder 114 of the edge device.
[0059] In step S406, the image prompt is input to the visual prompt image encoder 106 in the edge device to generate a visual representation. The visual transformer 102 is a deep learning model based on the transformer architecture and uses a self-attention mechanism to extract visual features from the image prompt. Unlike convolutional neural networks that rely on convolutional layers to extract local features from images, the visual transformer 102 uses a self-attention mechanism to capture global dependencies between image patches. This allows the visual transformer 102 to model the relationship between distant image regions and capture more complex visual patterns. However, such complex and dense visual features can fragment fine-grained image information and bring a large amount of computation due to the lengthy sequence.
[0060] To alleviate this problem, embodiments of the present invention employ a visual abstraction module 104, such as a query transformer (Q-Former). The query transformer is a lightweight transformer that employs a set of learnable query vectors to extract visual features from the visual cue image encoder 106. It acts as an information bottleneck between the visual cue image encoder 106 and the visual cue large language model 100 fused to the edge cloud. The query transformer extracts the most useful visual representation of language information while removing irrelevant visual information. In step S407, the visual representation is input to a fully connected linear projector 108 in the edge device to generate an image query embedding. The fully connected linear projector 108 is implemented by using a deep neural network (DNN) to transform the dimension from the visual representation to the same dimension as the text query embedding.
[0061] In step S408, the image query embedding is compared with the knowledge-based image embeddings of the second library in the large language model database 116 to find the top k most similar embeddings in the second library. Then, in step S408, the embedding that best matches the image query embedding is selected from the top k most similar embeddings and compared with the image query embedding to generate a second similarity score. If the second similarity score is greater than the second threshold, proceed to step S409. If the second similarity score is less than the second threshold, proceed to step S410. In step S409, the image query embedding is input to the small feature extraction model 110 to generate a second answer. The second answer is a quick response to the image query embedding and is implemented only by the large language model feature encoder 114 in the edge device.
[0062] In step S410, the text query embedding and the image query embedding are concatenated to generate a concatenated query embedding. Then, in step S412, the concatenated query embedding is compared with the concatenated knowledge-based embedding of the third library in the large language model database 116 to find the top k most similar embeddings in the third library. Then, the embedding that best matches the concatenated query embedding is selected from the top k most similar embeddings and compared with the concatenated query embedding to generate a third similarity score. If the third similarity score is greater than the third threshold, proceed to step S413. If the third similarity score is less than the third threshold, proceed to step S414. In step S413, the concatenated query embedding is input to the small feature extraction model 110 to generate a third answer. The third answer is a quick response to the concatenated query embedding and is implemented only by the large language model feature encoder 114 in the edge device. In step S414, the concatenated query embedding is input to the large feature extraction model 112 in the cloud to generate a fourth answer.
[0063] In an embodiment of the present invention, the large language model database 116 is initialized using the embedding results output by the large feature extraction model 112 in the cloud. All machine learning models except the large feature extraction model 112 are trained in the cloud and run in the edge device. The large feature extraction model 112 is trained and run in the cloud. This model achieves efficient resource utilization and low latency by adopting an edge-cloud collaborative architecture to balance local computing and cloud computing. Simple answers can be replied in the edge device, and complex answers can be replied in the cloud. In addition, the visual representation is converted to the same dimension as the text embedding, so that the large language model 100 provides accurate responses based on visual cues fused from the edge cloud.
[0064] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the claims of the present invention should fall within the protection scope of the present invention.
Claims
1. A method for performing edge-cloud fusion of visually cued large language models, characterized in that include: Training a large language model feature encoder and a small feature extraction model in an edge device; inputting a knowledge-based textual prompt into the large language model feature encoder in the edge device to generate a plurality of knowledge-based textual embeddings; Building a large language model database in the edge device based on the knowledge-based text embedding; Inputting a text prompt into the large language model feature encoder to generate a text query embedding; comparing the text query embedding to the large language model database to generate a first similarity score; and If the first similarity score is greater than a first threshold, the text query embedding is input into the small feature extraction model to generate a first answer.
2. The method according to claim 1, characterized in that Training the large language model feature encoder and the small feature extraction model is to train the large language model feature encoder and the small feature extraction model in a cloud.
3. The method according to claim 1, characterized in that Also included is applying a first database in the edge device to the large language model database; Wherein comparing the text query embedding with the large language model database to generate the first similarity score comprises: comparing the text query embedding with the first database to find k most similar embeddings in the first database; Selecting an embedding from the k most similar embeddings that best matches the text query embedding; and The textual query embedding is compared to the embedding that best matches the textual query embedding to generate the first similarity score.
4. A method for performing edge-cloud fusion of visually cued large language models, characterized in that include: Training a visual cue image encoder, a fully connected linear projector, and a small feature extraction model in an edge device; inputting a plurality of knowledge-based image cues into the visual cue image encoder in the edge device to generate a plurality of knowledge-based image embeddings; Building a large language model database in the edge device based on the knowledge-based image embedding; inputting an image cue into the visual cue image encoder to generate a visual representation; Inputting the visual representation into the fully connected linear projector in the edge device to generate an image query embedding; comparing the image query embedding to the large language model database to generate a second similarity score; and If the second similarity score is greater than a second threshold, the image query embedding is input into the small feature extraction model to generate a second answer.
5. The method according to claim 4, characterized in that Training the visual cue image encoder, the fully connected linear projector, and the small feature extraction model is to train the visual cue image encoder, the fully connected linear projector, and the small feature extraction model in a cloud.
6. The method according to claim 4, characterized in that Also included is applying a second database in the edge device to the large language model database; Wherein comparing the image query embedding with the large language model database to generate the second similarity score comprises: comparing the image query embedding with the second database to find k most similar embeddings in the second database; Selecting an embedding from the k most similar embeddings that best matches the image query embedding; and The image query embedding is compared to the embedding that best matches the image query embedding to generate the second similarity score.
7. The method according to claim 4, characterized in that The visual cue image encoder includes a visual transformer.
8. The method according to claim 7, characterized in that Also includes: The visual transformer is trained to extract visual features from the image cue using a transformer architecture and a self-attention mechanism.
9. The method according to claim 8, characterized in that The visual cue image encoder includes a visual abstraction module.
10. The method according to claim 9, characterized in that The visual abstraction module is a query transformer.
11. The method according to claim 10, characterized in that Also includes: The visual features are input into the query transformer to extract a useful visual representation of linguistic information while removing irrelevant visual information.
12. A method for performing edge-cloud fusion of visually cued large language models, characterized in that include: Training a visual cue image encoder, a fully connected linear projector, a large language model feature encoder, and a small feature extraction model in an edge device; Inputting a plurality of knowledge-based image cues and a plurality of knowledge-based text cues into the visual cue image encoder and the large language model feature encoder in the edge device, respectively, to generate a plurality of knowledge-based image embeddings and a plurality of knowledge-based text embeddings; concatenating the knowledge-based image embedding and the knowledge-based text embedding to generate a plurality of knowledge-based concatenated embeddings; Building a large language model database embedded in the edge device based on the knowledge-based connection; inputting an image cue into the visual cue image encoder in the edge device to generate a visual representation; Inputting the visual representation to the fully connected linear projector in the edge device to generate an image query embedding; Inputting a text prompt into the large language model feature encoder in the edge device to generate a text query embedding; concatenating the image query embedding and the text query embedding to generate a concatenated query embedding; comparing the concatenated query embedding to the large language model database to generate a third similarity score; and If the third similarity score is greater than a third threshold, the concatenated query embedding is input into the small feature extraction model to generate a third answer.
13. The method according to claim 12, characterized in that Training the visual cue image encoder, the fully connected linear projector, the large language model feature encoder and the small feature extraction model is to train the visual cue image encoder, the fully connected linear projector, the large language model feature encoder and the small feature extraction model on a cloud.
14. The method according to claim 12, characterized in that Also included is applying a third database in the edge device to the large language model database; Wherein comparing the concatenated query embedding with the large language model database to generate the third similarity score comprises: comparing the concatenated query embedding with the third database to find k most similar embeddings in the third database; Selecting an embedding from the k most similar embeddings that best matches the connected query embedding; and The connected query embedding is compared to the embedding that best matches the connected query embedding to generate the third similarity score.
15. The method according to claim 12, characterized in that The visual cue image encoder includes a visual transformer.
16. The method according to claim 15, characterized in that Also includes: The visual transformer is trained to extract multiple visual features from the image cue by using a transformer architecture and a self-attention mechanism.
17. The method according to claim 16, characterized in that The visual cue image encoder includes a visual abstraction module.
18. The method according to claim 17, characterized in that The visual abstraction module is a query transformer.
19. The method according to claim 18, characterized in that Also includes: The visual features are input into the query transformer to extract a useful visual representation of linguistic information while removing irrelevant visual information.
20. The method according to claim 12, characterized in that Also includes: If the third similarity score is less than the third threshold, the concatenated query embedding is input into a large feature extraction model to generate a fourth answer.