Large model distributed reasoning acceleration method based on multi-modal feature fusion and dynamic weight optimization
Through the multimodal feature fusion and dynamic weight optimization methods, the problems of waste of computing resources and increased energy consumption in multi-node deployment are solved, efficient cache resource sharing and load balancing are achieved, and the inference efficiency and system performance of the large model are improved.
Patent Information
- Application Number
- CN202510958489.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-11
- Publication Date
- 2025-08-29
AI Technical Summary
When using conventional load balancing strategies for multi-node deployment, there are problems such as wasting computing resources, increasing system energy consumption and reduced request processing speed. Especially in high concurrency business scenarios, repeated computing requirements lead to wasting computing resources and degradation of system performance.
A large-model distributed inference method based on multimodal feature fusion and dynamic weight optimization is adopted. Real-time semantic analysis is performed through the embedded inference context analysis module, cache adaptability is calculated, and decision-making is made based on node load and network quality, and global cache directory services are deployed to realize semantic-level context awareness and cache resource sharing.
It significantly improves inference efficiency and system throughput, reduces redundant computing and energy consumption, improves cache hit rate and request processing speed, and ensures load balancing and efficient resource utilization of computing nodes.
Smart Images

Figure CN120560855A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of large model technology, and more specifically, to a large model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization. Background Art
[0002] With the continuous advancement of technology, the scale of models in cutting-edge fields such as deep learning continues to expand, and the number of model parameters has increased dramatically. This trend has led to a significant increase in the demand for graphics memory during the inference phase. However, with current hardware conditions, the memory capacity of a single graphics card is no longer sufficient to meet the data storage and computing requirements of large-scale model inference. To address this issue, parallel inference architectures, featuring multiple graphics cards on a single machine and multiple machines with multiple graphics cards, have emerged as the mainstream solution for ensuring smooth inference of large-scale models. This architecture fully utilizes the computing resources of multiple graphics cards or multiple machines to collaboratively complete model inference tasks, alleviating graphics memory pressure and significantly improving inference efficiency.
[0003] At the same time, model services deployed on a single node face significant challenges in high-concurrency business scenarios. Due to the sheer volume of concurrent requests, a single node's computing resources and processing power struggle to respond to all requests quickly enough, increasing service response latency and potentially even causing service crashes. To address this challenge, a multi-node deployment solution is often adopted. This distributed architecture distributes requests to different nodes for processing, improving the system's overall throughput and concurrent processing capabilities.
[0004] However, conventional load balancing strategies (such as the Nginx-based round-robin algorithm) for multi-node deployments still have certain shortcomings. These strategies primarily focus on balancing request distribution, ensuring roughly equal load across nodes, but neglect the effective use of node-level caches for computational results. During model inference, some requests may require repeated computations. If each node independently processes all requests without sharing or caching computational results, a large amount of repeated computation will result. This not only wastes computing resources and increases system energy consumption, but also slows down request processing, impacting the user experience. Summary of the Invention
[0005] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization, which is used to solve the problems of wasting computing resources, increasing the overall energy consumption of the system, and reducing the request processing speed when adopting conventional load balancing strategies for multi-node deployment.
[0006] The technical solution adopted by the present invention is a large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization, which includes the following steps:
[0007] S1: Real-time semantic parsing is performed to obtain semantic features through the reasoning context analysis module embedded in the load balancer;
[0008] S2: Calculate cache fitness based on semantic features;
[0009] S3: Query the global cache directory service to obtain a list of matching nodes. It then uses a multi-fusion decision-making algorithm to make a decision based on cache adaptation, node load, and network quality parameters.
[0010] S4: Deploy a global cache directory service at the load balancing layer based on the decision, maintain a local cache pool on each computing node, and synchronize cache metadata to the global cache directory service in real time.
[0011] During inference, the inference context analysis module embedded in the load balancer can parse the semantic features of the input in real time, achieving semantic-level context awareness. This upgrades traditional URL / IP-based routing to a semantic feature-driven one, preventing similar requests from being scattered across different nodes, fully utilizing the cache, and accelerating inference. Cache adaptability is calculated based on these semantic features, enabling the system to more accurately identify the computing requirements and cache adaptability of different requests, providing more accurate data support for caching decisions. This allows the system to adjust caching strategies based on request characteristics, optimize computing resource usage, and effectively reduce redundant computation and energy consumption. The decision-making process also considers multiple factors, such as node load and network quality. This ensures load balancing across computing nodes, maximizes cache hit rates, reduces data transmission latency, and dynamically adjusts computing resource allocation across nodes to accommodate high-concurrency requests in different business scenarios. Finally, a global cache directory service is deployed at the load balancing layer, allowing each computing node to maintain a local cache pool and synchronize cache metadata to the global cache directory service in real time. This ensures efficient sharing of cache resources across computing nodes, avoids duplicate computation, and significantly improves inference efficiency and system throughput.
[0012] Preferably, step S1 includes constructing a feature vector, and calculating semantic similarity using a two-stage fusion algorithm based on the constructed feature vector, wherein the feature vector includes a length feature and a topic hash; the semantic similarity calculation formula is:
[0013] sim(Q,S k )=α×sim lenght +β×sim topic
[0014] Among them, sim(Q,S k ) represents the request Q and cache entry S kSemantic similarity of sim lenght Indicates length similarity, sim topic Indicates topic similarity, α and β represent sim lenght and sim topic The weight coefficient of .
[0015] In this application, a method for calculating the similarity between the query vector and the historical cache data is designed. The length feature and the topic hash are integrated to effectively capture the multi-dimensional information of the input data, making the calculation of semantic similarity more accurate, significantly improving the accuracy and efficiency of semantic parsing, realizing the efficient fusion of multimodal features, and improving the semantic understanding ability in the reasoning process.
[0016] Preferably, the length feature is constructed by counting the number of input tokens and normalizing them to the interval [0, 1]. The calculation formula is:
[0017]
[0018] Among them, L Q Indicates the current request length; Represents the average processing length of cache entry records; ∈ represents an anti-zero constant.
[0019] Using the above formula to construct length features helps to effectively measure the complexity and information content of the input data. The normalization process can eliminate the impact of different input lengths on the processing of large models, so that large models have a consistent scale when processing inputs of various lengths. In addition, the introduction of length features can also provide additional contextual information for semantic analysis, which helps the model to more accurately allocate computing resources during inference. Through this method, large models can better balance computing efficiency and accuracy during inference, improve inference speed and optimize computing load, thereby achieving more efficient inference acceleration.
[0020] Preferably, the subject hash is constructed by generating a 16-bit subject identification code using an improved SimHash algorithm, and the calculation formula is:
[0021]
[0022] Wherein, δ(a, b) represents the matching indicator function. When a=b, δ(a, b) is 1, and when a≠b, δ(a, b) is 0. Q i Indicates the i-th binary value of the hash of the current request subject; S k,i Represents the i-th binary value of the subject hash of the k-th cache entry.
[0023] By using an improved SimHash algorithm to generate 16-bit topic identification codes to construct topic hashes, the processing efficiency and accuracy of text data can be significantly improved. The improved SimHash algorithm can extract stable features based on the semantic content of the text and generate hash values with high discrimination. When constructing the topic hash, the 16-bit topic identification code can compress the high-dimensional features of the text into a low-dimensional representation, reducing the storage and computational burden while maintaining the key features of the semantic information. This method not only improves the accuracy of text clustering and similarity calculations, but also demonstrates strong robustness and processing speed in large-scale data processing scenarios, making it suitable for fast and effective semantic analysis and data retrieval tasks.
[0024] Preferably, in step S2, the calculation formula of the cache adaptability is:
[0025]
[0026] Among them; C i Represents cache adaptability; ω k Represents dynamic weight; Indicates the dynamic generation time attenuation factor; F k represents cache freshness; H represents cache hit history correlation; n represents the total number of candidate cache entries for the current node; k represents the loop index variable of the cache entry; d represents decay, emphasizing that this factor is the quantitative result of time decay; γ represents the adjustment coefficient of historical hit correlation.
[0027] In this application, a cache matching calculation is designed, and a semantic similarity decay function is introduced to take into account factors such as query vector similarity, time window, and cache hit history correlation. This enables the system to more accurately identify the computing requirements and cache adaptability of different requests, providing more accurate data support for subsequent cache decisions.
[0028] Preferably, the method includes parameter tuning of the dynamically generated time decay factor and introducing compensation for high-similarity old caches; the calculation formula of the dynamically generated time decay factor is:
[0029] T d =e -λΔt
[0030] Among them, λ represents the attenuation coefficient; Δt represents the current time and the cache entry S k The time difference between the last update and the last update.
[0031] In this application, by tuning the time decay parameters and implementing the similarity compensation mechanism to introduce compensation for high-similarity old caches, the cache adaptability and cache hit rate can be significantly improved, making the calculated cache adaptation more accurate, thereby further providing more accurate data support for subsequent cache decisions.
[0032] Preferably, the cache freshness is positively correlated with the cache update frequency, and the calculation formula for the cache freshness is:
[0033]
[0034] Among them, age k Represents cache entry S k The length of time since the last update, in seconds; max_age indicates the maximum time the cache entry is allowed to be retained, in seconds. If it exceeds this time, it will be forced to expire.
[0035] Introducing a parameter that positively correlates cache freshness with cache update frequency into the cache fitness formula can significantly improve the system's accuracy in evaluating cache effectiveness. When cache freshness and update frequency are positively correlated, large models can more sensitively identify and prioritize frequently updated and highly fresh cached data during distributed reasoning, ensuring that large models always maintain an efficient caching mechanism in rapidly changing environments, reducing the impact of outdated data on calculation results, and improving cache accuracy and response speed. Specifically, this positive correlation promotes timely updates to cached data, avoids redundant storage of outdated data, and ensures that cached content remains consistent with current needs, thereby improving model performance, especially in high-concurrency and large-scale data processing scenarios.
[0036] Preferably, the calculation formula of the cache hit history correlation is:
[0037]
[0038] in, Represents cache entry S k At time t k The number of hits; t k Indicates the timestamp of the k-th hit; γ represents the decay coefficient, which controls the decay rate of the historical hit weight.
[0039] By introducing cache hit history correlation, the large model can adjust the cache priority based on past access patterns and cache hit situations. This historical correlation parameter helps identify which data has been frequently hit in past requests, thereby predicting its future access probability. Based on this historical data, the cache content can be dynamically optimized, the cache hit rate can be improved, and the number of cache misses can be reduced.
[0040] Preferably, in step S3, the use of a multi-fusion decision algorithm to make a decision includes using a nonlinear fusion strategy to calculate the final weight to obtain the optimal node route, and the calculation formula is:
[0041]
[0042] Among them, W i represents the final weight; C i Indicates cache adaptability; L i represents the node load factor; B i represents the network quality factor; β represents the network quality adjustment coefficient.
[0043] A multi-dimensional nonlinear weight evaluation function is designed to balance cache efficiency, computing load and network quality, solving the indicator coupling problem that traditional linear weighting cannot handle. It can perform weighted calculations based on the complex nonlinear relationship between different parameters, avoiding the oversimplified judgments that may be brought about by linear algorithms, making decisions more in line with the complexity of actual applications, and making more intelligent and efficient decisions, thereby improving the overall performance and reliability of the system.
[0044] Preferably, the step S4 includes:
[0045] S41: Perform incremental metadata synchronization, using a lightweight heartbeat protocol to synchronize node cache metadata to the global cache directory service every 5 seconds;
[0046] S42: Use Kernel Bypass technology to establish an inter-node RDMA high-speed channel for direct memory-to-memory data transmission;
[0047] S43: Use an asynchronous verification mechanism to ensure data consistency, record cache changes through distributed transaction logs, and perform data backup in combination with regular snapshots. When a failure occurs, the backup data will be restored.
[0048] In step S4, data synchronization, transmission efficiency, data consistency, and system fault tolerance are combined and optimized to improve the performance, stability, and reliability of the overall system. Through low-latency high-speed data channels, real-time synchronization, and efficient backup and recovery mechanisms, the system can maintain efficient operation in high-load, high-concurrency, and complex environments, reducing the impact of failures and ensuring uninterrupted business. This solves the problems of ignoring the use of node-level calculation result caches when using conventional load balancing strategies for multi-node deployment, and the possibility of partially repeated calculation requirements for different requests during model inference, resulting in a large amount of repeated calculations, a waste of computing resources, increased overall system energy consumption, and a reduction in request processing speed.
[0049] Compared with the prior art, the present invention has the following beneficial effects:
[0050] 1. Implement semantic-level context awareness, upgrading traditional URL / IP-based routing to semantic feature-driven routing. This prevents similar requests from being scattered across different nodes, fully utilizes cache, and accelerates inference.
[0051] 2. Design a similarity calculation method between the query vector and historical cache data, integrating length features and topic hashes to achieve efficient fusion of multimodal features;
[0052] 3. We designed a cache matching calculation, introduced a semantic similarity decay function, and took into account factors such as query vector similarity, time window, and cache hit history correlation.
[0053] 4. Design a multi-dimensional nonlinear weight evaluation function to balance cache efficiency, computational load, and network quality, solving the indicator coupling problem that traditional linear weighting cannot handle. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 This is a flowchart of the large-model distributed reasoning acceleration method provided by the present invention.
[0055] Figure 2 This is a schematic diagram of the characteristic vector provided by the present invention.
[0056] Figure 3 This is a flowchart of the semantic similarity calculation provided by the present invention.
[0057] Figure 4 This is a calculation flow chart of the multi-fusion decision algorithm provided by the present invention. DETAILED DESCRIPTION
[0058] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting the present invention. To better illustrate the following embodiments, some components in the accompanying drawings may be omitted, enlarged, or reduced in size, and do not represent actual product dimensions. Those skilled in the art will appreciate that some well-known structures and their descriptions may be omitted from the accompanying drawings.
[0059] Example 1
[0060] like Figure 1 As shown, this embodiment provides a large model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization, the method comprising the following steps:
[0061] Step S1: Real-time semantic parsing is performed by the reasoning context analysis module embedded in the load balancer to obtain semantic features;
[0062] Preferably, if Figure 3 As shown, Figure 3The flowchart of semantic similarity calculation provided in this embodiment is as follows: in step S1 of this embodiment, when a client request enters the Nginx load balancer, real-time semantic analysis is first performed through the embedded inference context analysis module (ICAM), and a feature vector is constructed. The semantic similarity is calculated based on the constructed feature vector using a two-stage fusion algorithm, wherein, Figure 2 As shown, the feature vector includes a length feature and a topic hash;
[0063] The semantic similarity calculation formula is:
[0064] sim(Q,S k )=α×sim lenght +β×sim topic
[0065] Among them, sim(Q,S k ) represents the request Q and cache entry S k The semantic similarity of , its range is [0,1]; sim lenght Indicates length similarity, sim topic Indicates topic similarity, α and β represent sim lenght and sim topic The weight coefficients are as follows, wherein α and β satisfy α+β=1. In this embodiment, α is set to 0.3 and β is set to 0.7.
[0066] In this embodiment, a method for calculating the similarity between the query vector and the historical cached data is designed, which integrates the length feature and the topic hash, can effectively capture the multi-dimensional information of the input data, make the calculation of semantic similarity more accurate, significantly improve the accuracy and efficiency of semantic parsing, realize the efficient fusion of multimodal features, and improve the semantic understanding ability in the reasoning process.
[0067] Specifically, in this embodiment, the length feature is constructed by calculating the number of input tokens and normalizing them to the interval [0, 1]. The calculation formula is:
[0068]
[0069] Among them, L Q Indicates the current request length; Represents the average processing length of cache entry records; ∈ represents an anti-zero constant.
[0070] Using the above formula to construct length features helps to effectively measure the complexity and information content of the input data. The normalization process can eliminate the impact of different input lengths on the processing of large models, so that large models have a consistent scale when processing inputs of various lengths. In addition, the introduction of length features can also provide additional contextual information for semantic analysis, which helps the model allocate computing resources more accurately during inference. Through this method, large models can better balance computing efficiency and accuracy during inference, improve inference speed and optimize computing load, thereby achieving more efficient inference acceleration.
[0071] In addition, in this embodiment, a 16-bit subject identification code is generated by using an improved SimHash algorithm to construct a subject hash, and the calculation formula is:
[0072]
[0073] Wherein, δ(a, b) represents the matching indicator function. When a=b, δ(a, b) is 1, and when a≠b, δ(a, b) is 0. Q i Indicates the i-th binary value of the hash of the current request subject; S k,i Represents the i-th binary value of the subject hash of the k-th cache entry.
[0074] Therefore, in this embodiment, by using the improved SimHash algorithm to generate a 16-bit topic identification code to construct a topic hash, the processing efficiency and accuracy of text data can be significantly improved. The improved SimHash algorithm can extract stable features based on the semantic content of the text and generate a hash value with high discrimination. When constructing the topic hash, the 16-bit topic identification code can compress the high-dimensional features of the text into a low-dimensional representation, reducing the storage and computational burden while maintaining the key features of the semantic information. This method not only improves the accuracy of text clustering and similarity calculations, but also demonstrates strong robustness and processing speed in large-scale data processing scenarios, making it suitable for fast and effective semantic analysis and data retrieval tasks.
[0075] Step S2: Calculate cache fitness based on semantic features;
[0076] Preferably, in step S2, the calculation formula of the cache adaptability is:
[0077]
[0078] Among them; C i Represents cache adaptability; ω k Represents dynamic weight; Indicates the dynamic generation time attenuation factor; F krepresents cache freshness; H represents cache hit history correlation; n represents the total number of candidate cache entries for the current node; k represents the loop index variable of the cache entry; d represents decay, emphasizing that this factor is the quantitative result of time decay; γ represents the adjustment coefficient of historical hit correlation.
[0079] In this embodiment, by designing cache matching calculation and introducing a semantic similarity decay function, factors such as query vector similarity, time window, and cache hit history correlation are taken into account, enabling the system to more accurately identify the computing requirements and cache adaptability of different requests, providing more accurate data support for subsequent cache decisions.
[0080] Preferably, the method includes parameter tuning of the dynamically generated time decay factor and introducing compensation for high-similarity old caches; the calculation formula of the dynamically generated time decay factor is:
[0081] T d =e -λΔt
[0082] Among them, λ represents the attenuation coefficient; Δt represents the current time and the cache entry S k The time difference between the last update and the last update.
[0083] In this embodiment, the parameter tuning is performed by dynamically generating the time decay factor, and the original parameter λ=0.05 / second is adjusted to λ=0.02 / second, T d =exp(-0.02×Δt), thus effectively extending the cache time by adjusting the λ value;
[0084] In addition, the adjustment is performed by introducing compensation for the high-similarity old cache. In this embodiment, if sim(Q,S k )>0.8 and age<10min, then ω′ k =1.2, thus increasing the initial weight of the new cache entry by 20%. Therefore, in this embodiment, by tuning the time decay parameters and implementing a similarity compensation mechanism to compensate for high-similarity old caches, the cache adaptability and cache hit rate are significantly improved, making the calculated cache fitness more accurate, thereby further providing more accurate data support for subsequent caching decisions.
[0085] Preferably, the cache freshness is positively correlated with the cache update frequency, and the calculation formula for the cache freshness is:
[0086]
[0087] Among them, age k Represents cache entry S kThe length of time since the last update, in seconds; max_age indicates the maximum time the cache entry is allowed to be retained, in seconds. If it exceeds this time, it will be forced to expire.
[0088] Introducing a parameter that positively correlates cache freshness with cache update frequency into the cache fitness formula can significantly improve the system's accuracy in evaluating cache effectiveness. When cache freshness and update frequency are positively correlated, large models can more sensitively identify and prioritize frequently updated and highly fresh cached data during distributed reasoning, ensuring that large models always maintain an efficient caching mechanism in rapidly changing environments, reducing the impact of outdated data on calculation results, and improving cache accuracy and response speed. Specifically, this positive correlation promotes timely updates to cached data, avoids redundant storage of outdated data, and ensures that cached content remains consistent with current needs, thereby improving model performance, especially in high-concurrency and large-scale data processing scenarios.
[0089] Preferably, the calculation formula of the cache hit history correlation is:
[0090]
[0091] in, Represents cache entry S k At time t k The number of hits; t k The timestamp of the kth hit; γ represents the decay coefficient, which controls the decay rate of the historical hit weight. In this embodiment, the window period of H is 60 minutes, and γ is set to 0.1.
[0092] By introducing cache hit history correlation, the large model can adjust the cache priority based on past access patterns and cache hit situations. This historical correlation parameter helps identify which data has been frequently hit in past requests, thereby predicting its future access probability. Based on this historical data, the cache content can be dynamically optimized, the cache hit rate can be improved, and the number of cache misses can be reduced.
[0093] Step S3: Query the global cache directory service to obtain a list of matching nodes, and use a multi-fusion decision algorithm to make a decision based on the cache adaptation, node load, and network quality parameters.
[0094] Preferably, in step S3, as Figure 4 As shown, Figure 4 The multi-fusion algorithm decision calculation flow chart provided in this embodiment includes: constructing a three-dimensional weight evaluation system, and then using a nonlinear fusion strategy to calculate the final weight to obtain the optimal node route. The calculation formula is:
[0095]
[0096] Among them, W i represents the final weight; C i Indicates cache adaptability; L i represents the node load factor; B i represents the network quality factor; β represents the network quality adjustment coefficient.
[0097] In this embodiment, the node load factor is calculated in real time by the GPU memory occupancy rate, and its calculation formula is:
[0098] L i =0.6*U mem +0.4*U core
[0099] Among them, U mem Indicates GPU memory usage (video memory usage / total video memory); U core Indicates GPU core utilization (the percentage of actual work performed by the computing unit).
[0100] The network quality factor is a dynamic measurement of the RDMA bandwidth between nodes, with a period of 200ms. The default value of the network quality adjustment coefficient is 0.15.
[0101] Therefore, in this embodiment, by designing a multi-dimensional nonlinear weight evaluation function, balancing cache efficiency, computing load and network quality, the indicator coupling problem that traditional linear weighting cannot handle is solved, and weighted calculations can be performed based on the complex nonlinear relationship between different parameters, avoiding the oversimplified judgments that may be brought about by linear algorithms, so that decisions are more in line with the complexity of actual applications, and more intelligent and efficient decisions can be made, thereby improving the overall performance and reliability of the system.
[0102] Step S4: Deploy the Global Cache Directory Service (GCDS) at the load balancing layer based on the decision, maintain a local cache pool at each computing node, and synchronize cache metadata to the Global Cache Directory Service in real time.
[0103] Preferably, the step S4 includes:
[0104] Step S41: perform incremental metadata synchronization, using a lightweight heartbeat protocol to synchronize node cache metadata to the global cache directory service every 5 seconds;
[0105] Step S42: establishing an inter-node RDMA high-speed channel through Kernel Bypass technology for direct data transmission from video memory to video memory;
[0106] Step S43: An asynchronous verification mechanism is used to ensure data consistency, and cache changes are recorded through a distributed transaction log (WAL). Data is backed up in combination with periodic snapshots. When a failure occurs, the backed-up data is restored. In this embodiment, the periodic snapshot is set to perform data backup every 10 minutes.
[0107] In step S4, data synchronization, transmission efficiency, data consistency, and system fault tolerance are combined and optimized to improve the performance, stability, and reliability of the overall system. Through low-latency high-speed data channels, real-time synchronization, and efficient backup and recovery mechanisms, the system can maintain efficient operation in high-load, high-concurrency, and complex environments, reducing the impact of failures and ensuring uninterrupted business. This solves the problems of ignoring the use of node-level calculation result caches when using conventional load balancing strategies for multi-node deployment, and the possibility of partially repeated calculation requirements for different requests during model inference, resulting in a large amount of repeated calculations, a waste of computing resources, increased overall system energy consumption, and a reduction in request processing speed.
[0108] Specifically, suppose that at a certain moment the load balancer needs to calculate the weights of three candidate nodes. The relevant parameters are shown in Table 1:
[0109] node <![CDATA[C i ]]> <![CDATA[L i ]]> <![CDATA[B i ]]> Node A 0.85 0.6 25 Node B 0.72 0.4 40 Node C 0.53 0.75 18
[0110] Table 1
[0111] Based on the above table, the parameter definitions and value ranges are as follows:
[0112] C i Cache adaptability: A comprehensive score based on semantic matching and cache freshness, ranging from [0, 1]. A value of 1.0 indicates a perfect match and the cache is up-to-date; a value of 0.0 indicates no valid cache.
[0113] L i Node load factor: reflects the current computing pressure of the node, ranging from [0,1]. A value of 1.0 indicates complete idleness; a value of 0.0 indicates full load.
[0114] B i Indicates the network quality factor: RDMA network effective bandwidth (unit: Gbps), range [10,200].
[0115] Then use W according to the data in the table i Formula for calculation, Among them, β = 0.15.
[0116] First calculate tanh(C i ):
[0117] For node A, tanh(0.85)≈0.691; node B: tanh(0.72)≈0.621; node C: tanh(0.53)≈0.487.
[0118] Then calculate (1-σ(L i ))(Sigmoid function),
[0119]
[0120] For node A: 1-σ(0.6)≈0.355, node B: 1-σ(0.4)≈0.402, node C: 1-σ(0.75)≈0.321.
[0121] Then calculate
[0122] For node A: e -0.15 / 25 ≈0.994, node B: e -0.15 / 40 ≈0.996, node C: e -0.15 / 18 ≈0.992.
[0123] Finally, the comprehensive weight calculation is performed, and the results are shown in Table 2.
[0124]
[0125]
[0126] Table 2
[0127] Based on the above results, parameter sensitivity analysis was performed, that is, the impact of single variable changes when other parameters are fixed. The obtained analysis results are shown in Table 3.
[0128] parameter Range of variation Weight change range Sensitivity level <![CDATA[C i ]]> 0.5→0.9 +72% <![CDATA[L i ]]> 0.3→0.7 -41% <![CDATA[B i ]]> 10→50Gbps +18%
[0129] Table 3
[0130] Therefore, according to the above conclusions, the system will give priority to high C i Nodes with low load are selected first, followed by nodes with good network communication.
[0131] Finally, during the inference process of this embodiment, the inference context analysis module embedded in the load balancer can parse the semantic features of the input in real time, realize semantic-level context awareness, upgrade the traditional URL / IP-based routing to semantic feature-driven, avoid similar requests being scattered to different nodes, make full use of the cache, speed up the inference speed, and calculate the cache adaptability based on these semantic features, so that the system can more accurately identify the computing requirements and cache adaptability of different requests, provide more accurate data support for cache decisions, and thus adjust the cache strategy according to the characteristics of the request, optimize the use of computing resources, and effectively reduce redundant computing and energy consumption. In addition, the decision-making process takes into account multiple factors such as node load and network quality, which not only ensures load balancing among computing nodes, but also maximizes the cache hit rate, reduces data transmission delay, and can dynamically adjust the computing resource allocation of each node to cope with high concurrency requests in different business scenarios. Finally, a global cache directory service is deployed through the load balancing layer, so that each computing node can maintain a local cache pool and synchronize cache metadata to the global cache directory service in real time, ensuring that computing nodes can efficiently share cache resources and avoid repeated computing, thereby significantly improving inference efficiency and system throughput.
[0132] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included within the scope of protection of the claims of the present invention.
Claims
1. A large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization, characterized in that: The method comprises the following steps: S1: Real-time semantic parsing is performed to obtain semantic features through the reasoning context analysis module embedded in the load balancer; S2: Calculate cache fitness based on semantic features; S3: Query the global cache directory service to obtain a list of matching nodes. It then uses a multi-fusion decision-making algorithm to make a decision based on cache adaptation, node load, and network quality parameters. S4: Deploy a global cache directory service at the load balancing layer based on the decision, maintain a local cache pool on each computing node, and synchronize cache metadata to the global cache directory service in real time.
2. The large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization according to claim 1 is characterized in that: Step S1 includes constructing a feature vector and calculating semantic similarity using a two-stage fusion algorithm based on the constructed feature vector, wherein the feature vector includes a length feature and a topic hash; the semantic similarity calculation formula is: sim(Q,S k )=α×sim lenght +β×sim topic Among them, sim(Q,S k ) represents the request Q and cache entry S k Semantic similarity of sim lenght Indicates length similarity, sim topic Indicates topic similarity, α and β represent sim lenght and sim topic The weight coefficient of .
3. The large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization according to claim 2 is characterized in that: The length feature is constructed by counting the number of input tokens and normalizing them to the interval [0,1]. The calculation formula is: Among them, L Q Indicates the current request length; L Sk Represents the average processing length of cache entry records; ∈ represents an anti-zero constant.
4. The large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization according to claim 2 is characterized in that: The subject hash is constructed by generating a 16-bit subject identification code using the improved SimHash algorithm, and its calculation formula is: Wherein, δ(a, b) represents the matching indicator function. When a=b, δ(a, b) is 1, and when a≠b, δ(a, b) is 0. Q i Indicates the i-th binary value of the hash of the current request subject; S k,i Represents the i-th binary value of the subject hash of the k-th cache entry.
5. The large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization according to claim 2 is characterized in that: In step S2, the calculation formula of the cache adaptability is: in; C i Represents cache adaptability; ω k Represents dynamic weight; Indicates the dynamic generation time attenuation factor; F k represents cache freshness; H represents cache hit history correlation; n represents the total number of candidate cache entries for the current node; k represents the loop index variable of the cache entry; d represents decay, emphasizing that this factor is the quantitative result of time decay; γ represents the adjustment coefficient of historical hit correlation.
6. The large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization according to claim 5 is characterized in that: This includes parameter tuning of the dynamically generated time decay factor and introducing compensation for high-similarity old caches. The calculation formula for the dynamically generated time decay factor is: T d =e -λΔt Among them, λ represents the attenuation coefficient; Δt represents the current time and the cache entry S k The time difference between the last update.
7. The large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization according to claim 5 is characterized in that: The cache freshness is positively correlated with the cache update frequency. The calculation formula for the cache freshness is: Among them, age k Represents cache entry S k The length of time since the last update, in seconds; max_age indicates the maximum time the cache entry is allowed to be retained, in seconds. If it exceeds this time, it will be forced to expire.
8. The large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization according to claim 5 is characterized in that: The calculation formula of the cache hit history correlation is: in, Represents cache entry S k At time t k The number of hits; t k Indicates the timestamp of the k-th hit; γ represents the decay coefficient, which controls the decay rate of the historical hit weight.
9. The large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization according to claim 5 is characterized in that: In step S3, the multi-fusion decision algorithm is used to make a decision, which includes using a nonlinear fusion strategy to calculate the final weight to obtain the optimal node route. The calculation formula is: Among them, W i represents the final weight; C i Indicates cache adaptability; L i represents the node load factor; B i represents the network quality factor; β represents the network quality adjustment coefficient.
10. The large-model distributed inference acceleration method based on multimodal feature fusion and dynamic weight optimization according to claim 1, characterized in that: The step S4 includes: S41: Perform incremental metadata synchronization, using a lightweight heartbeat protocol to synchronize node cache metadata to the global cache directory service every 5 seconds; S42: Use Kernel Bypass technology to establish an inter-node RDMA high-speed channel for direct memory-to-memory data transmission; S43: Use an asynchronous verification mechanism to ensure data consistency, record cache changes through distributed transaction logs, and perform data backup in combination with regular snapshots. When a failure occurs, the backup data will be restored.
Citation Information
Cited By
Large model reasoning acceleration method, device and equipment
CN120952186A
High-concurrency lightweight data channel adaptive load balancing method based on large model
CN120980082A
Mixed retrieval result LLM reordering method based on polling allocation
CN121328572A
