Token transmission optimization method for edge cloud collaboration of large model

Abstract

The invention discloses a Token transmission optimization method for edge cloud collaboration of a large model, and the method comprises the following steps: S1, deploying a backbone network of the large model in a cloud server, and decomposing the large model into a plurality of first expert sub-models hierarchically, and relates to the technical field of real-time communication of a generative large model in an industrial edge-cloud collaboration scene. According to the method, real-time confidence analysis is performed on the Token sequence generated by edge preliminary reasoning, and only the Token with low confidence is packaged into the Token-FEC frame to be transmitted to the cloud, so that the transmission data volume of a cloud-side link is reduced. Meanwhile, the Token-FEC frame adopts a 24-byte fixed structure, and the effective data length is defined in combination with a 2-bit TokenSize field, so that invalid data transmission redundancy is relieved, the bandwidth utilization rate is further improved, and the problems of low bandwidth utilization rate and high transmission redundancy caused by inference of an intermediate result due to complete transmission in a cloud-edge coordination scheme in a traditional scheme are solved.

Description

Technical Field

[0001] This invention relates to the field of real-time communication technology for generative large models in industrial edge-cloud collaborative scenarios, and particularly to a token transmission optimization method for large model edge-cloud collaboration. Background Technology

[0002] With the widespread application of generative large models (models with ≥10 billion parameters) in industrial scenarios, edge-cloud collaborative computing has become a key paradigm supporting real-time intelligent decision-making. However, deploying large-scale models on resource-constrained edge devices faces severe challenges: edge devices have limited memory, power consumption, and computing power, making it impossible to support complete large model inference tasks, while traditional cloud-edge collaborative solutions suffer from high transmission latency and low bandwidth utilization, making it difficult to meet the millisecond-level real-time response requirements in industrial control scenarios.

[0003] Publication number CN101656731B discloses a reliability control protocol based on forward error correction, but the above application has the following problems: On the one hand, the lack of processing of intermediate result tokens for large model inference leads to high transmission redundancy and large latency; on the other hand, the failure to dynamically adjust the redundancy strategy in combination with network status makes it easy for data loss to occur, resulting in unstable end-to-end latency from the inference request initiated by the industrial terminal to the output result, which cannot meet the real-time response requirements of industrial scenarios. Summary of the Invention

[0004] To address the technical problems existing in the background art, this invention proposes an optimized token transmission method for large-scale model edge cloud collaboration.

[0005] This invention proposes a token transmission optimization method for large-scale edge cloud collaboration, comprising the following steps: S1. Deploy the backbone network of a large model with no less than 10 billion parameters on a cloud server, and decompose the large model into multiple first-expert sub-models according to the hierarchy. On the edge device side, several second expert sub-models can be pre-cached based on historical access frequency and industrial scenario requirements; S2. After the industrial terminal equipment initiates a real-time inference request, the edge device calls several second expert sub-models to perform preliminary inference on the request and generate an initial output token sequence; real-time confidence analysis is performed on the initial output token sequence, and tokens whose confidence does not meet the preset threshold are marked as low-confidence tokens. S3. Encapsulate the low-confidence token into a Token-FEC frame and transmit it to the cloud server; S4. The cloud server receives the Token-FEC frame, parses out the low-confidence Token, and then performs secondary inference on it by the main network of the large model and the relevant first expert sub-model to generate a high-confidence corrected Token. The corrected Token is then re-encapsulated into a Token-FEC frame and sent to the edge device. S5. The edge device receives the corrected token, and merges the corrected token with the high-confidence token to generate the final inference result.

[0006] Preferably, in S2, real-time confidence analysis is performed on the initial output token sequence, and tokens whose confidence does not meet a preset threshold are marked as low-confidence tokens, as follows: Obtain the information entropy of the probability distribution corresponding to the initial output token, and mark the tokens with entropy values ​​higher than a preset threshold as low confidence tokens.

[0007] Preferably, in S3, the size of the Token-FEC frame is 24 bytes, and the Token-FEC frame includes a header and a payload; The header includes a sequence number, RoPE rotation angle, and CRC check field. The header is used to ensure the reliability of transmission and the consistency of the model context; the payload is used to carry the token data to be transmitted.

[0008] Preferably, in S3, on the edge device side, r redundant FEC packets are generated for each group of Token-FEC frames to be sent, and the r redundant FEC packets are sent to the cloud server along with the Token-FEC frames; the purpose is that the receiving end can use these r redundant packets to recover the lost data packets in this group without waiting for retransmission. The value of r is generated in the following way: Get the network round-trip time (RTT) of the current network, and get the average time interval (Ts) from the generation to the transmission of a single Token-FEC frame in the current network; Where r = ⌈K·(Ts / RTT)⌉, K is a preset scaling factor, the value of K is (0,1), RTT≠0, and r is a positive integer.

[0009] Preferably, in S4, when the cloud server receives a Token-FEC frame from the edge device, it records the reception status of the Token-FEC frame using a preset first bit diagram; when the cloud server determines that the Token-FEC frame from the edge device is lost through CRC check or sequence number continuity check, it sends a retransmission request for the lost Token-FEC frame to the edge device; the edge device responds to the retransmission request and retransmits the requested Token-FEC frame to the cloud server. When the edge device receives a Token-FEC frame from the cloud server, it records the reception status of the Token-FEC frame using a preset second bitmap. When the edge device determines that the Token-FEC frame from the cloud server is lost through CRC check or sequence number continuity check, it sends a retransmission request for the lost Token-FEC frame to the cloud server. The cloud server responds to the retransmission request and retransmits the requested Token-FEC frame to the edge device.

[0010] Preferably, in S3, the header of the Token-FEC frame also includes a 2-bit TokenSize field, which is used to define the effective length of the Token in the payload. As an explanation, the purpose of the TokenSize field is to inform the receiving end, such as a cloud server or edge device, how many bytes in the fixed-byte payload are actual and valid token data.

[0011] The proposed token transmission optimization method for large-scale edge cloud collaboration in this invention has the following beneficial technical effects: 1. This application performs real-time confidence analysis on the token sequence generated by preliminary inference at the edge, encapsulating only low-confidence tokens into Token-FEC frames for transmission to the cloud, thereby reducing the amount of data transmitted in the cloud-edge link. Simultaneously, the Token-FEC frame adopts a fixed 24-byte structure, combined with a 2-bit TokenSize field defining the effective data length, alleviating redundant invalid data transmission and further improving bandwidth utilization. This addresses the problem of low bandwidth utilization and high transmission redundancy in traditional cloud-edge collaborative solutions caused by transmitting complete intermediate inference results. Through a collaborative mode of preliminary inference on the edge device side plus secondary inference on the cloud server, it helps achieve the dual goals of reducing edge computing power and increasing cloud-edge transmission efficiency while ensuring inference accuracy.

[0012] 2. This application addresses the issue of current technologies being unable to adapt to the transmission characteristics of industrial scenarios. Firstly, it balances transmission reliability and model context consistency by using the sequence number, CRC check field, and RoPE rotation angle in the Token-FEC frame header. Secondly, it dynamically calculates the number of redundant FEC packets based on the current network round-trip time (RTT) and the average time interval (Ts) from generation to transmission of a single Token-FEC frame. This adapts the transmission to the jitter characteristics of 5G uRLLC links, allowing the receiver to directly recover lost data using redundant packets without waiting for retransmission. Thirdly, it uses a bitmap to record the Token-FEC frame reception status, combined with CRC check and sequence number continuity checks to trigger retransmission and prevent data loss. This makes the end-to-end latency from the industrial terminal device initiating an inference request to the output result more stable, meeting the real-time response requirements of industrial control scenarios.

[0013] 3. This application deploys the large model backbone network in the cloud and decomposes it into first expert sub-models in a hierarchical manner. At the same time, it only pre-caches second expert sub-models selected based on historical access frequency and industrial scenario requirements on the edge device side. This alleviates the pressure on the edge device to carry out the complete large model inference task, and can reduce the memory occupation and computing load of the edge device, so that large-scale generative models can be deployed on resource-constrained industrial edge devices. Attached Figure Description

[0014] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0016] like Figure 1 The proposed method for optimizing token transmission in large-scale edge cloud collaboration includes the following steps: S1. Deploy the backbone network of a large model with no less than 10 billion parameters on a cloud server, and decompose the large model into multiple first-expert sub-models according to the hierarchy. On the edge device side, several second expert sub-models can be pre-cached based on historical access frequency and industrial scenario requirements; This application deploys a large model backbone network in the cloud and decomposes it into a first expert sub-model in a hierarchical manner. At the same time, it pre-caches only the second expert sub-model, which is selected based on historical access frequency and industrial scenario requirements, on the edge device side. This alleviates the pressure on the edge device to carry out the inference task of the complete large model, and reduces the memory occupation and computing load of the edge device, so that large-scale generative models can be deployed on resource-constrained industrial edge devices.

[0017] S2. After the industrial terminal equipment initiates a real-time inference request, the edge device calls several second expert sub-models to perform preliminary inference on the request and generate an initial output token sequence; real-time confidence analysis is performed on the initial output token sequence, and tokens whose confidence does not meet the preset threshold are marked as low-confidence tokens. S3. Encapsulate the low-confidence token into a Token-FEC frame and transmit it to the cloud server; S4. The cloud server receives the Token-FEC frame, parses out the low-confidence Token, performs secondary inference on it through the first expert sub-model, and generates a high-confidence corrected Token; the corrected Token is re-encapsulated into a Token-FEC frame and sent to the edge device. As an explanation, after the cloud server receives the Token-FEC frame and parses out the low-confidence Token, the large model backbone network performs secondary inference on it. After the large model backbone network obtains the low-confidence Token, it dynamically routes it to one or more first expert sub-models through the internal control network for secondary inference to generate the corrected Token. S5. The edge device receives the corrected token, and merges the corrected token with the high-confidence token to generate the final inference result.

[0018] This application performs real-time confidence analysis on the token sequence generated by preliminary inference at the edge, encapsulating only low-confidence tokens into Token-FEC frames for transmission to the cloud, thereby reducing the amount of data transmitted in the cloud-edge link. Simultaneously, the Token-FEC frame adopts a fixed 24-byte structure, combined with a 2-bit TokenSize field defining the effective data length, alleviating redundant data transmission of invalid data and further improving bandwidth utilization. This addresses the problem of low bandwidth utilization and high transmission redundancy in traditional cloud-edge collaborative solutions caused by transmitting complete intermediate inference results. Through a collaborative mode of preliminary inference on the edge device side plus secondary inference on the cloud server, it helps achieve the dual goals of reducing edge computing costs and increasing cloud-edge transmission efficiency while ensuring inference accuracy.

[0019] In an optional embodiment, in S2, a real-time confidence analysis is performed on the initial output token sequence, and tokens whose confidence does not meet a preset threshold are marked as low-confidence tokens, as follows: Obtain the information entropy of the probability distribution corresponding to the initial output token, and mark the tokens with entropy values ​​higher than a preset threshold as low confidence tokens.

[0020] In an optional embodiment, in S3, the Token-FEC frame is 24 bytes in size and includes a header and a payload. The header includes a sequence number, RoPE rotation angle, and CRC check field. The header is used to ensure the reliability of transmission and the consistency of the model context; the payload is used to carry the token data to be transmitted.

[0021] In an optional embodiment, the Token-FEC frame is 24 bytes in size, the header is 8 bytes in size, and the payload is 16 bytes in size. On the edge device side, r redundant FEC packets are generated for each group of Token-FEC frames to be sent. The r redundant FEC packets are sent to the cloud server along with the Token-FEC frames. The purpose is that the receiving end can use these r redundant packets to recover the lost data packets in this group without waiting for retransmission. The value of r is generated in the following way: Get the network round-trip time (RTT) of the current network, and get the average time interval (Ts) from the generation to the transmission of a single Token-FEC frame in the current network; Where r = ⌈K·(Ts / RTT)⌉, K is a preset scaling factor, the value of K is (0,1), RTT≠0, and r is a positive integer.

[0022] This application addresses the issue of current technologies being unable to adapt to the transmission characteristics of industrial scenarios. Firstly, it balances transmission reliability and model context consistency by using the sequence number, CRC check field, and RoPE rotation angle in the Token-FEC frame header. Secondly, it dynamically calculates the number of redundant FEC packets based on the current network round-trip time (RTT) and the average time interval (Ts) from generation to transmission of a single Token-FEC frame. This adapts the transmission to the jitter characteristics of 5G uRLLC links, allowing the receiver to directly recover lost data using redundant packets without waiting for retransmission. Thirdly, it uses a bitmap to record the Token-FEC frame reception status, combined with CRC check and sequence number continuity checks to trigger retransmission and prevent data loss. This results in a more stable end-to-end latency from the initiation of an inference request by the industrial terminal device to the output result, meeting the real-time response requirements of industrial control scenarios.

[0023] In an optional embodiment, in S4, when the cloud server receives a Token-FEC frame from the edge device, it records the reception status of the Token-FEC frame using a preset first bit diagram; when the cloud server determines that the Token-FEC frame from the edge device is lost through CRC check or sequence number continuity check, it sends a retransmission request for the lost Token-FEC frame to the edge device; the edge device responds to the retransmission request and retransmits the requested Token-FEC frame to the cloud server. When the edge device receives a Token-FEC frame from the cloud server, it records the reception status of the Token-FEC frame using a preset second bitmap. When the edge device determines that the Token-FEC frame from the cloud server is lost through CRC check or sequence number continuity check, it sends a retransmission request for the lost Token-FEC frame to the cloud server. The cloud server responds to the retransmission request and retransmits the requested Token-FEC frame to the edge device.

[0024] In an optional embodiment, in S3, the header of the Token-FEC frame also includes a 2-bit TokenSize field, which is used to define the effective length of the Token in the payload. As an explanation, the purpose of the TokenSize field is to inform the receiving end, such as a cloud server or edge device, how many bytes in the fixed-byte payload are actual and valid token data.

[0025] By caching only the high-frequency expert sub-model on the edge device side through S1, the memory usage of the edge device is reduced. The bandwidth utilization of the cloud-edge link is improved through the dynamic redundancy FEC packets in S3 and the frame-by-frame retransmission mechanism in S4. For illustrative purposes, the large model refers to existing industrial visual inspection models or industrial predictive maintenance models.

[0026] For clarification, "acquisition" in this application refers to obtaining the required content or data using existing technical means.

[0027] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0028] In the embodiments provided by this invention, it should be understood that the disclosed system or method can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for instance, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.

[0029] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0030] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.

[0031] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the basic characteristics of the present invention.

[0032] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A token transmission optimization method for large-scale model edge cloud collaboration, characterized in that, Includes the following steps: S1. Deploy the backbone network of the large model on a cloud server and decompose the large model into multiple first-expert sub-models according to the hierarchy; pre-cache several second-expert sub-models on the edge device side; S2. After the industrial terminal equipment initiates a real-time inference request, the edge device side calls several second expert sub-models to perform preliminary inference on the request and generate an initial output Token sequence. Real-time confidence analysis is performed on the initial output token sequence. Tokens whose confidence does not meet the preset threshold are marked as low-confidence tokens, and others are marked as high-confidence tokens. S3. Encapsulate the low-confidence token into a Token-FEC frame and transmit it to the cloud server; S4. The cloud server receives the Token-FEC frame, parses out the low-confidence Token, performs secondary inference on it through the first expert sub-model, and generates the corrected Token. The corrected Token is then re-encapsulated into a Token-FEC frame and sent to the edge device. S5. The edge device receives the corrected token, and merges the corrected token with the high-confidence token to generate the final inference result.

2. The token transmission optimization method for large-scale model edge cloud collaboration according to claim 1, characterized in that, In S2, real-time confidence analysis is performed on the initial output token sequence, and tokens whose confidence does not meet the preset threshold are marked as low-confidence tokens, as follows: Obtain the information entropy of the probability distribution corresponding to the initial output token, and mark the tokens with entropy values ​​higher than a preset threshold as low confidence tokens.

3. The token transmission optimization method for large-scale model edge cloud collaboration according to claim 1, characterized in that, In S3, the size of a Token-FEC frame is 24 bytes, and the Token-FEC frame includes a header and a payload; The header includes a serial number, RoPE rotation angle, and CRC check field; the payload carries the token data to be transmitted.

4. The token transmission optimization method for large-scale model edge cloud collaboration according to claim 3, characterized in that, In S3, on the edge device side, r redundant FEC packets are generated for each group of Token-FEC frames to be sent, and the r redundant FEC packets are sent to the cloud server along with the Token-FEC frames. The value of r is generated in the following way: Get the network round-trip time (RTT) of the current network, and get the average time interval (Ts) from the generation to the transmission of a single Token-FEC frame in the current network; Where r = ⌈K·(Ts / RTT)⌉, K is a preset scaling factor, the value of K is (0,1), RTT≠0, and r is a positive integer.

5. The token transmission optimization method for large-scale model edge cloud collaboration according to claim 1, characterized in that, In S4, when the cloud server receives a Token-FEC frame from the edge device, it records the reception status of the Token-FEC frame using a preset first bit diagram. When the cloud server determines that the Token-FEC frame from the edge device is lost through CRC check or sequence number continuity check, it sends a retransmission request for the lost Token-FEC frame to the edge device. The edge device responds to the retransmission request and retransmits the requested Token-FEC frame to the cloud server. When the edge device receives a Token-FEC frame from the cloud server, it records the reception status of the Token-FEC frame using a preset second bitmap. When the edge device determines that a Token-FEC frame from the cloud server is lost through CRC check or sequence number continuity check, it sends a retransmission request for the lost Token-FEC frame to the cloud server; the cloud server responds to the retransmission request and retransmits the requested Token-FEC to the edge device.

6. The token transmission optimization method for large-scale model edge cloud collaboration according to claim 1, characterized in that, In S3, the header of the Token-FEC frame also includes a 2-bit TokenSize field, which is used to define the valid length of the token in the payload.

Images