Response speed increasing system and method based on cross-hardware collaborative virtualization

Abstract

The invention discloses a response speed increasing system and method based on cross-hardware collaborative virtualization, and belongs to the technical field of heterogeneous hardware collaborative computing and virtualization. Through a unified virtual instruction set (VISA), distributed task scheduling and algebraic layout optimization, heterogeneous hardware differences are shielded, tasks are dynamically allocated, and communication delay is reduced. Experimental results show that according to the scheme, the end-to-end response time is shortened from 1-2 seconds to 0.5-1 second, the communication delay is reduced by 70%-80%, the hardware resource utilization rate is increased to 80% or above, and the cooperative computing efficiency and the real-time interaction performance are remarkably optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of heterogeneous hardware collaborative computing and virtualization, and specifically relates to a response speed improvement system and method based on cross-hardware collaborative virtualization. Background Art

[0002] With the rapid development of artificial intelligence and edge computing, the demand for deploying large language models (LLMs) on smart devices (such as smartphones, speakers, and in-vehicle systems) is increasing. However, edge devices and cloud servers are typically equipped with heterogeneous hardware (such as CPUs, GPUs, NPUs, and TPUs), with significant differences in computing power and power consumption, resulting in inefficient task allocation and data communication.

[0003] In the prior art, heterogeneous hardware collaboration solutions (such as the CUDA architecture, NVIDIA Guide v12.0) rely on hard-coded kernels and are only compatible with NVIDIA GPUs. They have poor versatility and lack dynamic task allocation capabilities, resulting in idle computing resources and high response delays. Hardware abstraction layer technologies (such as the Ladder system, OSDI 2024) achieve a certain degree of hardware compatibility through static priority scheduling, but are unable to adapt to dynamic loads, and the resource utilization of edge devices is usually less than 40%. In addition, existing solutions often use full data transmission in data distribution without considering network bandwidth fluctuations, resulting in large fluctuations in communication delays, which seriously affects real-time interactive performance. The present invention overcomes the above-mentioned defects through a unified virtual instruction set, dynamic task scheduling, and algebraic layout optimization, significantly improving the efficiency of collaborative computing. Summary of the Invention

[0004] In response to the problems of low efficiency of heterogeneous hardware collaboration, inflexible task allocation and high communication delay in the existing technology, the present invention proposes a response speed improvement system and method based on cross-hardware collaborative virtualization. The design is scientific and reasonable, overcomes the shortcomings of the existing technology, and has a significant performance improvement effect.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A response speed improvement system based on cross-hardware collaborative virtualization includes an edge device layer, a communication layer, and a server layer; wherein,

[0007] The edge device layer integrates a virtual instruction set parsing module, a task scheduling module, and a data distribution module, and is configured to perform task parsing and preprocessing; wherein,

[0008] A virtual instruction set parsing module configured to parse an input task into hardware-independent instructions;

[0009] A task scheduling module is configured to dynamically allocate tasks based on task complexity and hardware characteristics;

[0010] A data distribution module is configured to partition and prioritize data through an algebraic layout system and a dependency analysis algorithm based on graph theory, thereby optimizing a data distribution path;

[0011] The communication layer, which uses a bandwidth-aware scheduling protocol based on differential coding and incremental transmission technology and real-time bandwidth estimation, is configured for efficient data transmission and synchronization, and adapts to network fluctuations;

[0012] The server layer is configured with a collaborative computing module and a result output module; it is configured to perform complex computing tasks and return results; wherein,

[0013] A collaborative computing module configured to perform collaborative computing on heterogeneous hardware;

[0014] The result output module is configured to integrate the calculation results and return them to the edge device.

[0015] Preferably, the virtual instruction set parsing module supports dynamic instruction extension, adapts to heterogeneous hardware including CPU, GPU, NPU, and TPU through the instruction template library, and supports 8-bit or 16-bit low-precision fixed-point operations.

[0016] Preferably, the task scheduling module adopts a load balancing algorithm based on reinforcement learning to analyze the task computing amount, data dependency and hardware load status in real time and dynamically generate an allocation strategy.

[0017] Preferably, the algebraic layout system of the data distribution module includes:

[0018] A data sharding submodule is configured to shard data based on a graph theory dependency analysis algorithm;

[0019] a prioritization submodule, configured to sort the data shards by mission criticality;

[0020] The metadata generation submodule is configured to record fragmentation and priority information to optimize the transmission path.

[0021] Preferably, the incremental transmission technology of the communication layer only transmits the data changes, the bandwidth-aware scheduling protocol dynamically adjusts the transmission rate according to real-time network fluctuations, and includes an error retransmission mechanism.

[0022] In addition, the present invention also provides a method for improving response speed based on cross-hardware collaborative virtualization, which adopts the above-mentioned system for improving response speed based on cross-hardware collaborative virtualization, and is characterized in that it specifically includes the following steps:

[0023] Step 1: Perform task analysis through the virtual instruction set analysis module;

[0024] Through the virtual instruction set, the input task is parsed into a unified instruction, shielding the differences between heterogeneous hardware including CPU, GPU, NPU, and TPU;

[0025] Step 2: Dynamically allocate tasks through the task scheduling module;

[0026] Dynamically allocate tasks to the optimal hardware unit of edge devices or servers based on task complexity and hardware characteristics;

[0027] Step 3: The data distribution module segments and prioritizes the data to generate an optimized transmission path.

[0028] Utilizes an algebraic layout system to optimize data distribution paths between edge devices and servers through data sharding and prioritization.

[0029] Step 4: Perform incremental transmission and bandwidth adaptive synchronization through the communication layer;

[0030] Only changed data is transmitted, and the transmission strategy is dynamically adjusted in combination with bandwidth-aware scheduling to adapt to network fluctuations. An error retransmission mechanism is used to ensure data synchronization stability.

[0031] Step 5: Perform collaborative computing at the server layer and return the results to the edge device;

[0032] The collaborative computing module performs collaborative computing on heterogeneous hardware, and the result output module integrates the computing results and returns them to the edge device.

[0033] Preferably, in step 1, task parsing includes the following steps:

[0034] Step 1.1: Deploy a lightweight virtual machine and run the virtual instruction set parsing input task;

[0035] Step 1.2: Assign tasks to appropriate hardware based on task type.

[0036] For speech tasks, feature data extraction is assigned to low-power hardware; for image tasks, complex calculations are assigned to highly parallel hardware;

[0037] Step 1.3: Control virtual instruction set parsing complexity.

[0038] Preferably, step 2 specifically includes the following steps:

[0039] Step 2.1: Analyze task complexity and hardware load in real time and generate allocation strategies;

[0040] Step 2.2: Use a load balancing algorithm to allocate tasks, prioritizing tasks to idle or high-performance hardware;

[0041] Step 2.3: Adjust the allocation strategy based on the scenario.

[0042] Preferably, the data distribution optimization in step 3 includes the following steps:

[0043] Step 3.1: Use the algebraic layout system to partition and prioritize data based on graph-theoretic dependency analysis algorithms.

[0044] Step 3.2: Compress the data;

[0045] Step 3.3: Generate metadata to record sharding and priority information.

[0046] Preferably, step 5 includes the following steps:

[0047] Step 5.1: The server receives the optimized data and distributes it to the collaborative computing module;

[0048] Step 5.2: Perform collaborative computing on heterogeneous hardware through the collaborative computing module;

[0049] Step 5.3: Integrate the calculation results through the result output module and return high-priority data to the edge device first.

[0050] The beneficial technical effects brought about by the present invention are:

[0051] This invention significantly improves the collaborative computing efficiency of edge devices and servers through a unified virtual instruction set, dynamic task scheduling, and algebraic layout optimization; the details are as follows:

[0052] Improved response speed: End-to-end response time is shortened from 1-2 seconds to 0.5-1 second, a reduction of approximately 44%-50%;

[0053] Communication efficiency optimization: communication delay is reduced by 70%-80%, supporting low-bandwidth networks;

[0054] Improved resource utilization: Hardware utilization increased from less than 40% to 80%-85%;

[0055] Enhanced system flexibility: Through VISA dynamic instruction extensions, it supports new hardware types (such as ASICs and FPGAs), reducing system upgrade costs; it adapts to various application scenarios (such as voice interaction, image processing, in-vehicle AI, and the Internet of Things), improving system versatility;

[0056] Low power support: Optimizes edge device resource usage and enhances deployment capabilities in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of the structure of the cross-hardware collaborative virtualization module;

[0058] Figure 2Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0060] This paper proposes a response speed improvement solution based on cross-hardware collaborative virtualization. By designing a unified virtual instruction set (VISA), distributed task scheduling, and algebraic layout optimization, it achieves efficient collaboration between heterogeneous edge devices and server hardware. The main innovations are as follows:

[0061] 1. Virtual Unified Instruction Set (VISA): defines a hardware-independent virtual instruction set, shielding the differences between heterogeneous hardware such as CPU, GPU, NPU, TPU, etc., supporting low-precision calculations, and improving system compatibility.

[0062] 2. Distributed task scheduling: Dynamically allocate computing tasks based on task complexity and hardware characteristics to maximize resource utilization.

[0063] 3. Algebraic layout optimization: The algebraic layout system optimizes the efficient distribution of data between devices and servers, significantly reducing communication latency.

[0064] 4. Efficient synchronization mechanism: Incremental transmission and bandwidth-aware scheduling are used to ensure efficient and stable data synchronization.

[0065] The technical solution of the present invention includes the following core modules: Figure 1 As shown:

[0066] 1. Semantic-driven low-precision data processing module

[0067] (1) Function:

[0068] Through virtual instruction sets, distributed task scheduling, and data distribution optimization, collaborative computing of heterogeneous hardware is achieved, significantly improving response speed.

[0069] (2) Implementation methods, such as Figure 2 As shown:

[0070] 1) Virtual instruction set design:

[0071] Deploy lightweight virtual machines on edge devices and servers and define a hardware-independent virtual instruction set (VISA).

[0072] VISA supports low-precision calculations on CPUs, GPUs, NPUs, and TPUs, shielding underlying hardware differences and providing a unified task execution interface.

[0073] 2) Distributed task scheduling:

[0074] Design an intelligent scheduler to analyze task complexity (such as computational complexity and data dependency) and hardware characteristics (such as low power consumption of NPU and high parallelism of GPU) in real time.

[0075] Based on the analysis results, tasks are dynamically assigned to the optimal hardware unit to ensure load balancing.

[0076] 3) Data distribution optimization:

[0077] Utilize an algebraic layout system to optimize the data distribution path between edge devices and servers.

[0078] Through data segmentation and priority sorting, redundant transmission is reduced and communication delay is lowered.

[0079] 4) Efficient synchronization:

[0080] Adopting incremental transmission technology, only the changed data is transmitted, thus reducing bandwidth usage.

[0081] Combined with bandwidth-aware scheduling, data transmission strategies are dynamically adjusted to adapt to network fluctuations.

[0082] (3) Output:

[0083] Efficient task allocation solutions, synchronization of data and calculation results.

[0084] The following uses the voice interaction scenario of a smart speaker as an example to illustrate the specific implementation of the present invention:

[0085] Example 1: Real-time navigation using an in-vehicle AI system

[0086] 1. Scenario: The user enters the voice command "Plan the shortest route to the city center" into the in-vehicle AI system.

[0087] 2. Task analysis:

[0088] (1) The in-vehicle system is equipped with a CPU and NPU, and the server is equipped with a GPU.

[0089] (2) The Virtual Instruction Set (VISA) parses voice commands, assigns feature extraction tasks to the NPU, and assigns path planning tasks to the server GPU.

[0090] 3. Task Scheduling

[0091] The scheduler dynamically allocates tasks based on task complexity (speech processing is a lightweight task, path planning is a computationally intensive task) and hardware characteristics.

[0092] 4. Data distribution:

[0093] (1) The algebraic layout system optimizes the transmission of speech feature data (approximately 500KB) to the server, reducing the communication delay from 25ms to 6ms.

[0094] (2) Incremental transmission ensures that only key feature data is uploaded.

[0095] 5. Calculation and output:

[0096] (1) The server GPU completes the path planning and returns the navigation results.

[0097] (2) End-to-end response time is shortened from 1.6 seconds to 0.9 seconds.

[0098] 6. Results:

[0099] Response time was shortened by 44%, communication latency was reduced by 76%, and hardware resource utilization was increased to 82%.

[0100] Example 2: Smartphone Image Processing

[0101] 1. Scenario: A user takes a photo with a smartphone and requests to “optimize image brightness and contrast.”

[0102] 2. Task analysis:

[0103] (1) Smartphones are equipped with CPUs and GPUs, and servers are equipped with TPUs.

[0104] (2) The Virtual Instruction Set (VISA) parses image processing tasks, assigning preprocessing tasks (such as pixel sampling) to the mobile phone GPU and complex optimization tasks to the server TPU.

[0105] 3. Task Scheduling

[0106] The scheduler optimizes task allocation based on image resolution (high-resolution images) and hardware parallel capabilities.

[0107] 4. Data distribution:

[0108] (1) The algebraic layout system uses the JPEG compression algorithm to compress the preprocessed image data (about 2MB) to 400KB, reducing the transmission delay from 30ms to 7ms.

[0109] (2) Bandwidth-aware scheduling adapts to 4G network fluctuations.

[0110] 5. Calculation and output:

[0111] (1) The server TPU completes image optimization and returns the processed photo.

[0112] (2) End-to-end response time is shortened from 2 seconds to 1 second.

[0113] 6. Results:

[0114] Response time is shortened by 50%, communication latency is reduced by 77%, and hardware resource utilization is increased to 85%.

[0115] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A response speed improvement system based on cross-hardware collaborative virtualization, characterized in that: It includes edge device layer, communication layer and server layer; among them, The edge device layer integrates a virtual instruction set parsing module, a task scheduling module, and a data distribution module, and is configured to perform task parsing and preprocessing; wherein, A virtual instruction set parsing module configured to parse an input task into hardware-independent instructions; A task scheduling module is configured to dynamically allocate tasks based on task complexity and hardware characteristics; A data distribution module is configured to partition and prioritize data through an algebraic layout system and a dependency analysis algorithm based on graph theory, thereby optimizing a data distribution path; The communication layer, which uses a bandwidth-aware scheduling protocol based on differential coding and incremental transmission technology and real-time bandwidth estimation, is configured for efficient data transmission and synchronization, and adapts to network fluctuations; The server layer is configured with a collaborative computing module and a result output module; it is configured to perform complex computing tasks and return results; wherein, A collaborative computing module configured to perform collaborative computing on heterogeneous hardware; The result output module is configured to integrate the calculation results and return them to the edge device.

2. The response speed improvement system based on cross-hardware collaborative virtualization according to claim 1 is characterized in that: The virtual instruction set parsing module supports dynamic instruction extension, adapts to heterogeneous hardware including CPU, GPU, NPU, and TPU through the instruction template library, and supports 8-bit or 16-bit low-precision fixed-point operations.

3. The response speed improvement system based on cross-hardware collaborative virtualization according to claim 1 is characterized in that: The task scheduling module adopts a load balancing algorithm based on reinforcement learning to analyze the task computing volume, data dependency and hardware load status in real time and dynamically generate allocation strategies.

4. The response speed improvement system based on cross-hardware collaborative virtualization according to claim 1 is characterized in that: An algebraic layout system for data distribution modules, including: A data sharding submodule is configured to shard data based on a graph theory dependency analysis algorithm; a prioritization submodule, configured to sort the data shards by mission criticality; The metadata generation submodule is configured to record fragmentation and priority information to optimize the transmission path.

5. The response speed improvement system based on cross-hardware collaborative virtualization according to claim 1 is characterized in that: The incremental transmission technology of the communication layer only transmits the data changes. The bandwidth-aware scheduling protocol dynamically adjusts the transmission rate according to real-time network fluctuations and includes an error retransmission mechanism.

6. A method for improving response speed based on cross-hardware collaborative virtualization, using the system for improving response speed based on cross-hardware collaborative virtualization according to claim 1, characterized in that: The specific steps include: Step 1: Perform task analysis through the virtual instruction set analysis module; Through the virtual instruction set, the input task is parsed into a unified instruction, shielding the differences between heterogeneous hardware including CPU, GPU, NPU, and TPU; Step 2: Dynamically allocate tasks through the task scheduling module; Dynamically allocate tasks to the optimal hardware unit of edge devices or servers based on task complexity and hardware characteristics; Step 3: The data distribution module segments and prioritizes the data to generate an optimized transmission path. Utilizes an algebraic layout system to optimize data distribution paths between edge devices and servers through data sharding and prioritization. Step 4: Perform incremental transmission and bandwidth adaptive synchronization through the communication layer; Only changed data is transmitted, and the transmission strategy is dynamically adjusted in combination with bandwidth-aware scheduling to adapt to network fluctuations. An error retransmission mechanism is used to ensure data synchronization stability. Step 5: Perform collaborative computing at the server layer and return the results to the edge device; The collaborative computing module performs collaborative computing on heterogeneous hardware, and the result output module integrates the computing results and returns them to the edge device.

7. The response speed improvement method based on cross-hardware collaborative virtualization according to claim 6, characterized in that: In step 1, task parsing includes the following steps: Step 1.1: Deploy a lightweight virtual machine and run the virtual instruction set parsing input task; Step 1.2: Assign tasks to appropriate hardware based on task type. For speech tasks, feature data extraction is assigned to low-power hardware; for image tasks, complex calculations are assigned to highly parallel hardware; Step 1.3: Control virtual instruction set parsing complexity.

8. The response speed improvement method based on cross-hardware collaborative virtualization according to claim 6, characterized in that: Step 2 specifically includes the following steps: Step 2.1: Analyze task complexity and hardware load in real time and generate allocation strategies; Step 2.2: Use a load balancing algorithm to allocate tasks, prioritizing tasks to idle or high-performance hardware; Step 2.3: Adjust the allocation strategy based on the scenario.

9. The method for improving response speed based on cross-hardware collaborative virtualization according to claim 6, characterized in that: The data distribution optimization in step 3 includes the following steps: Step 3.1: Use the algebraic layout system to partition and prioritize data based on graph-theoretic dependency analysis algorithms. Step 3.2: Compress the data; Step 3.3: Generate metadata to record sharding and priority information.

10. The response speed improvement method based on cross-hardware collaborative virtualization according to claim 6, characterized in that: Step 5 includes the following steps: Step 5.1: The server receives the optimized data and distributes it to the collaborative computing module; Step 5.2: Perform collaborative computing on heterogeneous hardware through the collaborative computing module; Step 5.3: Integrate the calculation results through the result output module and return high-priority data to the edge device first.

Images