End-cloud collaborative visual language model reasoning method and system
By employing an edge-cloud collaborative architecture and channel-level 4-bit fixed-point quantization compression technology, the problems of insufficient computing power and privacy leakage of terminal devices are solved, enabling efficient and privacy-preserving visual language model services while reducing communication overhead and latency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-08
- Publication Date
- 2026-03-10
AI Technical Summary
In existing technologies, deploying small-sized visual language models on terminal devices results in insufficient computing power, while deploying large-sized models on the edge results in high power consumption. Furthermore, cloud deployment poses a risk of privacy leakage, and piecemeal deployment leads to huge communication overhead, making it difficult to achieve efficient and privacy-preserving visual language model services.
The edge-cloud collaborative architecture is adopted. The floating-point visual embedding tensor is compressed by channel-level 4-bit fixed-point quantization through edge devices. The compressed data packet is transmitted to the cloud for model inference, and then restored to a floating-point visual embedding tensor in the cloud for multimodal inference, thus achieving data compression and privacy protection.
It achieves efficient data compression (8x compression ratio), reduces network latency, maintains model performance, meets privacy protection requirements, simplifies deployment processes, and facilitates rapid integration and application.
Smart Images
Figure CN121638464A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, and relates to a visual language model reasoning method and system, especially an edge-cloud collaborative visual language model reasoning method and system. Background Technology
[0002] The integration of visual language models into terminal devices has become a hot topic in recent years. By leveraging the rich sensing devices on terminals, such as the front and rear cameras on mobile phones, the interior and exterior cameras in smart cars, and the cameras mounted on smart toys, visual language models can provide devices with richer functions and more interesting interactive experiences.
[0003] Generally, there are two approaches to connecting terminal devices to large visual language models: cloud deployment and edge deployment. Cloud deployment places the large visual language model in a computing center (i.e., in the cloud), typically running in a computing cluster equipped with high-end graphics cards. These high-performance computers can run flagship-level large visual language models, providing enhanced capabilities and interactive experiences. The terminal device sends images and instructions to the computing center via the network, waits for the results, and then presents them to the user. The advantage of this method is that it can use the largest possible visual language model; the disadvantage is that currently, directly sending images from the terminal device to the computing center raises compliance issues and privacy risks.
[0004] The edge deployment approach directly deploys a small-sized visual language model on the edge device, ensuring that images, commands, and question-and-answer responses all originate from the device. The most significant advantage of this method is privacy protection. However, the problem is that mainstream edge devices are primarily consumer-grade devices, whose computing chips cannot efficiently run large-sized visual language models; they can only run very small models. Furthermore, edge devices have extremely high power consumption requirements, making it impossible to provide the same service quality and user experience as cloud-based deployments.
[0005] Currently, some researchers have proposed deploying visual language models in chunks, such as deploying the visual compressor on the device side and the large text language model on the cloud side. However, this approach incurs huge communication overhead, far exceeding the communication required to transmit the original image, putting enormous pressure on both the device itself and the ingress bandwidth of the cloud service, making it difficult to implement practically at present.
[0006] Given the aforementioned shortcomings of existing technologies, there is an urgent need to research a novel edge-cloud collaborative visual language model reasoning method and system. Summary of the Invention
[0007] To overcome the shortcomings of existing technologies, this invention proposes a visual language model inference method and system that is edge-cloud collaborative. It can efficiently compress the floating-point visual embedding tensor generated at the edge without retraining the visual language model in the cloud.
[0008] To achieve the above objectives, the present invention provides the following technical solution: A visual language model inference method with edge-cloud collaboration, characterized by the following steps: S1. Obtain the input visual data through the edge device and convert the visual data into a floating-point visual embedding tensor; S2. Perform feature compression on the floating-point visual embedding tensor along each channel of the floating-point visual embedding tensor to obtain a compressed data packet; S3. Transmit the compressed data packet from the edge device to the cloud server via the network; S4. After receiving the compressed data packet, the cloud server performs feature dequantization on the compressed data packet to recover the recovered floating-point visual embedding tensor. S5. Input the recovered floating-point visual embedding tensor into the visual language model that is pre-deployed on the cloud server and kept frozen. Combine the text instructions input by the user, the visual language model performs multimodal reasoning and generates reasoning results. S6. The inference result is transmitted to the edge device via the network, and the edge device displays the inference result to the user.
[0009] Preferably, the step S1 of converting the visual data into a floating-point visual embedding tensor specifically includes: S11. Preprocess the visual data; S12. Input the preprocessed visual data into the visual encoder to convert the visual data into a high-dimensional feature vector; S13. Input the high-dimensional feature vector into the visual projection model to transform the high-dimensional feature vector into a floating-point visual embedding tensor.
[0010] Preferably, the preprocessing in step S11 includes: cropping, scaling, and interpolation.
[0011] Preferably, step S2 specifically includes: S21. For each channel, calculate its quantization parameters independently; S22. Apply the calculated quantization parameters to convert the floating-point visual embedding tensor linear mapping in the channel into a 4-digit integer value by rounding it down. S23. Pack all the 4-bit integer values of all channels and the corresponding quantization parameters of each channel into a compressed data packet.
[0012] Preferably, the quantization parameters in step S21 include a scaling factor and zeros, and scale[i] = (max(x[:, i]) – min(x[:, i])) / 15; offset[i] = round(-min(x[:, i]) / scale[i]); Where scale[i] is the scaling factor of the i-th channel, x[:, i] is the floating-point visual embedding tensor in the i-th channel, max(x[:, i]) is the maximum value of the floating-point visual embedding tensor in the i-th channel, min(x[:, i]) is the minimum value of the floating-point visual embedding tensor in the i-th channel, offset[i] is the zero point of the i-th channel, and round() means rounding to the nearest integer.
[0013] Preferably, in step S22, c[:, i] = round(x[:, i] / scale[i]) + offset[i], where c[:, i] is a 4-bit integer value of the i-th channel.
[0014] Preferably, the feature dequantization recovery of the compressed data packet in step S4 is specifically as follows: d[:, i] = (c[:, i] – offset[i]) * scale[i], where d[:, i] is the floating-point visual embedding tensor in the recovered i-th channel.
[0015] Furthermore, the present invention also provides an edge-cloud collaborative visual language model inference system, characterized in that it includes: A feature extraction module is used to acquire input visual data through an edge device and convert the visual data into a floating-point visual embedding tensor; A feature compression module is used to perform feature compression on the floating-point visual embedding tensor along each channel of the floating-point visual embedding tensor to obtain a compressed data packet; A data transmission module, used to transmit the compressed data packets from the edge device to the cloud server via a network; The feature dequantization recovery module is used to perform feature dequantization recovery on the compressed data packet after the cloud server receives the compressed data packet, so as to obtain the recovered floating-point visual embedding tensor. The model inference module is used to input the recovered floating-point visual embedding tensor into a visual language model that is pre-deployed on a cloud server and kept frozen. Combined with the text instructions input by the user, the visual language model performs multimodal inference and generates inference results. The results display module is used to transmit the inference results to an edge device via a network, and the edge device displays the inference results to the user.
[0016] Furthermore, the present invention also provides a visual language model inference device with edge-cloud collaboration, characterized in that it includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the edge-cloud collaborative visual language model inference method as described above. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the steps of the edge-cloud collaborative visual language model inference method as described above.
[0017] Compared with existing technologies, the edge-cloud collaborative visual language model reasoning method and system proposed in this invention have the following significant advantages: 1. Extremely high compression ratio: This invention achieves quantization from 32-bit floating-point numbers to 4-bit integers, theoretically achieving a data compression ratio of 8 times, greatly reducing the amount of data transmission, significantly reducing network latency, and making real-time interaction possible.
[0018] 2. Better maintain model performance: The channel-level (Per-Channel) quantization strategy, compared with the coarse-grained tensor-level (Per-Tensor) quantization, can handle the numerical distribution of different feature channels more finely, effectively reducing the information loss caused by quantization. Thus, while achieving a high compression ratio, it can maintain the original task accuracy of the visual language model to the greatest extent.
[0019] 3. Ensuring data privacy and compliance: This invention inherits the inherent advantages of the edge-cloud collaborative architecture, ensuring that the original visual data does not leave the local device, thus meeting strict privacy protection requirements.
[0020] 4. Low deployment cost and strong adaptability: This invention uses a post-training quantization (PTQ) technique, which does not require any retraining or fine-tuning of the visual language model. In particular, the expensive visual language model in the cloud can remain completely unchanged, which greatly simplifies the deployment process and facilitates rapid integration and application. Attached Figure Description
[0021] Figure 1 This is a flowchart of the visual language model reasoning method for edge-cloud collaboration of the present invention.
[0022] Figure 2 This is a schematic diagram of the structure of the edge-cloud collaborative visual language model reasoning system of the present invention. Detailed Implementation
[0023] Before detailing any embodiment of the invention, it should be understood that the invention, in its application, is not limited to the details of the construction and arrangement of the components set forth in the following description or illustrated in the following figures. The invention can have other embodiments and can be practiced or carried out in various ways. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “comprising” or “having” and variations thereof is intended to cover the items set forth below and their equivalents, as well as any additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “linkage,” and variations thereof are used broadly and cover both direct and indirect installation, connection, support, and linking. Moreover, “connection” and “linkage” are not limited to physical or mechanical connections or links. Furthermore, firstly, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention. Secondly, the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. The term "a" should not be construed as a limitation on the quantity.
[0024] In recent years, large language models have demonstrated outstanding capabilities in multimodal tasks, greatly improving the efficiency of human-computer interaction and the convenience of daily life. However, the large number of parameters and computational requirements of these powerful large language models limit their widespread deployment, especially on edge devices with limited computing power, memory, and power consumption (such as home vehicle systems, smartphones, and IoT devices).
[0025] To address this issue, a viable solution is to adopt an edge-cloud collaborative computing architecture: a relatively small visual encoder (currently ViT, or Visual Transformer), which does not require autoregressive computation, runs at the edge to extract features from images or videos. These features are then transmitted to the cloud, where a powerful Large Language Model (LLM) performs subsequent understanding, reasoning, and generation tasks. The advantage of this architecture is that the raw image or video data, which may contain privacy information, remains on the local device; only abstract features are uploaded, effectively resolving privacy compliance issues.
[0026] However, this architecture introduces a new technical bottleneck: communication overhead. While the visual embedding generated by the visual encoder is abstract, its data dimensionality is extremely high (e.g., a floating-point matrix containing hundreds of tokens, each with thousands of dimensions), resulting in a massive overall data volume. This leads to single-transmission data volumes reaching several megabytes. In applications with high real-time requirements (such as autonomous driving assistance and real-time question answering), transmitting this data causes severe network latency, significantly impacting the user experience.
[0027] Therefore, how to compress the visual embedding to be transmitted to the maximum extent while ensuring that the model performance is not significantly affected, and reduce the communication bandwidth and latency between the edge and the cloud, has become a key technical issue in realizing efficient and privacy-preserving visual language model (VLM) services.
[0028] To address the aforementioned issues, this invention proposes an edge-cloud collaborative visual language model inference method and system, which employs a visual embedding compression method based on per-channel 4-bit fixed-point quantization (Per-Channel INT4 Quantization). This method efficiently compresses floating-point visual embeddings generated at the edge without requiring retraining of the cloud-based visual language model.
[0029] Figure 1 A flowchart of the edge-cloud collaborative visual language model inference method of the present invention is shown. Figure 1 As shown, the edge-cloud collaborative visual language model inference method of the present invention includes the following steps: S1, Feature Extraction.
[0030] The input visual data is acquired through an edge device and converted into a floating-point visual embedding tensor.
[0031] Specifically, edge computing devices (such as home vehicle systems, smartphones, and IoT devices) acquire input visual data (such as images) through their equipped cameras. After preprocessing (including necessary cropping, scaling, and interpolation), the visual data is input into a visual encoder (a 300M-parameter Vision Transformer) to convert the visual data into a high-dimensional feature vector. This high-dimensional feature vector is then transformed into a visual embedding tensor, i.e., a floating-point visual embedding tensor, typically in FP32 or FP16 format, through a visual projection model (a 100M-parameter gated multilayer perceptron model).
[0032] S2, Feature compression.
[0033] The floating-point visual embedding tensor is subjected to feature compression along each channel to obtain a compressed data packet.
[0034] In this invention, the floating-point visual embedding tensor to be compressed is processed along its channel dimension.
[0035] Specifically, firstly, for each channel of the floating-point visual embedding tensor, its quantization parameters, namely the scaling factor and zero-point, are calculated independently. This ensures that each feature channel has a quantization range best suited to itself, preserving information to the greatest extent. The calculation formula is as follows: Assuming x represents the floating-point visual embedding tensor with a shape of S * H (S represents the dynamic sequence length, typically a number between 1024 and 32768, and H is the hidden layer size, which is 4096 in this invention), then the quantization parameters of the i-th channel are: scale[i] = (max(x[:, i]) – min(x[:, i])) / 15; offset[i] = round(-min(x[:, i]) / scale[i]); Where scale[i] is the scaling factor of the i-th channel, x[:, i] is the floating-point visual embedding tensor in the i-th channel, max(x[:, i]) is the maximum value of the floating-point visual embedding tensor in the i-th channel, min(x[:, i]) is the minimum value of the floating-point visual embedding tensor in the i-th channel, offset[i] is the zero point of the i-th channel, and round() means rounding to the nearest integer.
[0036] Secondly, using the calculated quantization parameters, the floating-point visual embedding tensor (which is a floating-point value) of this channel is linearly mapped and rounded to a 4-bit integer (INT4) value. The calculation formula is as follows: c[:, i] = round(x[:, i] / scale[i]) + offset[i]; Where c[:, i] is the 4-bit integer value of the i-th channel, that is, the INT4 value.
[0037] Then, the INT4 values of all channels (i.e., c[:, i]) and the corresponding quantization parameters of each channel (i.e., scale[i] and offset[i]) are packaged to form a compressed data packet.
[0038] This invention employs a per-channel quantization strategy, which, compared to the coarse-grained per-tensor quantization, can process the numerical distribution of different feature channels more finely, effectively reducing the information loss caused by quantization. Thus, while achieving a high compression ratio, it maximizes the maintenance of the original task accuracy of the visual language model.
[0039] S3: Data transmission.
[0040] The compressed data packets are transmitted from the edge device to the cloud server via the network.
[0041] After feature compression, the compressed data packets (including c[:, i], scale[i], and offset[i] for each channel) are transmitted from the edge device to the cloud server via the network to facilitate inference on the cloud server.
[0042] Because this invention achieves quantization from 32-bit floating-point numbers to 4-bit integers, it can theoretically achieve a data compression ratio of 8 times, greatly reducing the amount of data transmission and significantly reducing network latency, making real-time interaction possible.
[0043] S4: Feature inverse quantization recovery.
[0044] After receiving the compressed data packet, the cloud server performs feature dequantization on the compressed data packet to recover the recovered floating-point visual embedding tensor.
[0045] Specifically, after receiving the compressed data packet, the cloud server first parses out the INT4 value of each channel and the corresponding quantization parameters, namely, c[:, i], scale[i] and offset[i] of each channel.
[0046] Next, for each channel, using its corresponding scaling factor and zero point, the corresponding INT4 value is linearly mapped inversely to recover a floating-point visual embedding tensor. The calculation formula is as follows: d[:, i] = (c[:, i]– offset[i]) * scale[i]; Where d[:, i] is the floating-point visual embedding tensor in the recovered i-th channel.
[0047] S5: Model Inference.
[0048] The recovered floating-point visual embedding tensor is input into a visual language model that is pre-deployed on the cloud server and kept frozen. Combined with the text instructions input by the user, the visual language model performs multimodal reasoning and generates reasoning results.
[0049] This invention employs a post-training quantization (PTQ) technique, which eliminates the need for any retraining or fine-tuning of the visual language model (i.e., the visual language model remains frozen). In particular, visual language models deployed on cloud servers can remain completely unchanged, greatly simplifying the deployment process and facilitating rapid integration and application.
[0050] S6. Results Display.
[0051] The inference results are transmitted to an edge device via a network, and the edge device then displays the inference results to the user.
[0052] This invention adopts an edge-cloud collaborative approach, inheriting the inherent advantages of the edge-cloud collaborative architecture. The original visual data does not leave the local device (edge device), thus meeting strict privacy protection requirements.
[0053] Figure 2 A schematic diagram of the edge-cloud collaborative visual language model inference system of the present invention is shown. Figure 2 As shown, the edge-cloud collaborative visual language model inference system of the present invention includes: 1. Feature extraction module.
[0054] The feature extraction module is used to acquire input visual data through an edge device and convert the visual data into a floating-point visual embedding tensor.
[0055] 2. Feature compression module.
[0056] The feature compression module is used to perform feature compression on the floating-point visual embedding tensor along each channel of the floating-point visual embedding tensor to obtain a compressed data packet.
[0057] 3. Data transmission module.
[0058] The data transmission module is used to transmit the compressed data packets from the edge device to the cloud server via the network.
[0059] 4. Feature inverse quantization recovery module.
[0060] The feature dequantization recovery module is used to perform feature dequantization recovery on the compressed data packet after the cloud server receives the compressed data packet, so as to obtain the recovered floating-point visual embedding tensor.
[0061] 5. Model reasoning module.
[0062] The model inference module is used to input the recovered floating-point visual embedding tensor into a visual language model that is pre-deployed on a cloud server and kept frozen. Combined with the text instructions input by the user, the visual language model performs multimodal inference and generates inference results.
[0063] 6. Results display module.
[0064] The result display module is used to transmit the inference results to an edge device via a network, and the edge device displays the inference results to the user.
[0065] Furthermore, the present invention also provides an edge-cloud collaborative visual language model inference device, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the edge-cloud collaborative visual language model inference method as described above. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the edge-cloud collaborative visual language model inference method as described above.
[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention based on the concept of the present invention, without departing from the essence and scope of the technical solutions of the present invention.
Claims
1. An end-to-cloud collaborative visual language model inference method, characterized in that, The method comprises the following steps: S1, obtaining input visual data by an edge device and converting the visual data into a floating-point visual embedding tensor; S2, respectively performing feature compression on the floating-point visual embedding tensor along each channel of the floating-point visual embedding tensor to obtain a compressed data packet; S3, transmitting the compressed data packet from the edge device to a cloud server through a network; S4, after the cloud server receives the compressed data packet, performing feature dequantization recovery on the compressed data packet to obtain a restored floating-point visual embedding tensor; S5, inputting the restored floating-point visual embedding tensor into a visual language model pre-deployed on the cloud server and kept in a frozen state, combining a text instruction input by a user, performing multi-modal inference by the visual language model and generating an inference result; S6, transmitting the inference result to the edge device through a network, and displaying the inference result to the user by the edge device.
2. The end-to-cloud collaborative visual language model inference method according to claim 1, wherein, The conversion of the visual data into the floating-point visual embedding tensor in the step S1 specifically comprises: S11, preprocessing the visual data; S12, inputting the preprocessed visual data into a visual encoder to convert the visual data into a high-dimensional feature vector; S13, inputting the high-dimensional feature vector into a visual projection model to transform the high-dimensional feature vector into a floating-point visual embedding tensor.
3. The end-to-cloud collaborative visual language model inference method according to claim 2, wherein, The preprocessing in the step S11 comprises cropping, scaling and interpolation.
4. The end-to-cloud collaborative visual language model inference method according to any one of claims 1-3, characterized in that, The step S2 specifically comprises: S21, independently calculating the quantization parameter for each channel; S22, applying the calculated quantization parameter to linearly map and round the floating-point visual embedding tensor in the channel to convert it into a 4-bit integer value; S23, packing the 4-bit integer values of all channels and the corresponding quantization parameters of the channels to form a compressed data packet.
5. The end-to-cloud collaborative visual language model reasoning method according to claim 4, characterized in that, The quantization parameter in the step S21 comprises a scaling factor and a zero point, and scale[i] = (max(x[:, i]) – min(x[:, i])) / 15; offset[i] = round(-min(x[:, i]) / scale[i]); wherein scale[i] is the scaling factor of the i-th channel, x[:, i] is the floating-point visual embedding tensor in the i-th channel, max(x[:, i]) is the maximum value of the floating-point visual embedding tensor in the i-th channel, min(x[:, i]) is the minimum value of the floating-point visual embedding tensor in the i-th channel, offset[i] is the zero point of the i-th channel, and round() represents rounding to the nearest integer.
6. The end-to-cloud collaborative visual language model reasoning method according to claim 5, characterized in that, In the step S22, c[:, i] = round(x[:, i] / scale[i]) + offset[i], wherein c[:, i] is the 4-bit integer value of the i-th channel.
7. The end-to-cloud collaborative visual language model reasoning method according to claim 6, characterized in that, The feature dequantization recovery of the compressed data packet in the step S4 is specifically: d[:, i] = (c[:, i] - offset[i]) * scale[i], wherein d[:, i] is the recovered floating-point visual embedding tensor in the i-th channel.
8. An end-to-cloud collaborative visual language model inference system, comprising: Comprise: a feature extraction module for obtaining input visual data through an edge device and converting the visual data into a floating-point visual embedding tensor; a feature compression module for performing feature compression on the floating-point visual embedding tensor along each channel of the floating-point visual embedding tensor respectively to obtain a compressed data packet; a data transmission module for transmitting the compressed data packet from the edge device to a cloud server through a network; a feature dequantization recovery module for performing feature dequantization recovery on the compressed data packet after the cloud server receives the compressed data packet to obtain a recovered floating-point visual embedding tensor; a model inference module for inputting the recovered floating-point visual embedding tensor into a visual language model pre-deployed on the cloud server and kept in a frozen state, combining with a user input text instruction, performing multi-modal inference by the visual language model and generating an inference result; a result display module for transmitting the inference result to the edge device through a network and displaying the inference result to the user by the edge device.
9. An end-to-cloud collaborative visual language model inference device, comprising: Comprise: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the end-cloud collaborative visual language model inference method as claimed in any one of claims 1-7.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the end-cloud collaborative visual language model inference method as claimed in any one of claims 1-7.