Model lightweighting methods and electronic devices

Abstract

This application discloses a model lightweighting method and electronic device, relating to the field of artificial intelligence technology. The method includes dividing an input sequence into multiple data blocks; determining the routing vector of each data block based on its matching degree with multiple expert modules, thereby determining the target data block to be processed by each device, improving resource utilization; for any target data block to be processed by a remote device, determining the difference parameters of the target data block relative to shared basic parameters to generate a residual data block, obtaining an aggregated request packet; the remote device performs inference based on the aggregated request packet, reducing data transmission volume while ensuring inference accuracy; and determining the target inference result of the data block based on the inference result of the target expert module. This solves the technical problem in related technologies where methods for achieving large model lightweighting reduce inference accuracy, achieving the technical effect of ensuring inference accuracy while achieving large model lightweighting.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to methods for lightweighting models and electronic devices. Background Technology

[0002] As large models grow in size, their inference demands on hardware resources increase dramatically. Deploying large models on a single device is insufficient to meet the requirements for low-cost, low-latency deployment. To address this issue, related technologies utilize the deployment of large models across multiple devices to meet these deployment requirements.

[0003] When deploying large models across multiple devices, related technologies achieve lightweighting of large models through static model compression or static expert system techniques. However, the methods used in these technologies to achieve lightweighting of large models reduce their inference accuracy. Summary of the Invention

[0004] This application provides a model lightweighting method and electronic device to at least solve the problem that the methods of achieving lightweighting of large models in related technologies reduce the inference accuracy of large models.

[0005] This application provides a lightweight model method, including:

[0006] The input sequence of the target model is divided into multiple data blocks;

[0007] For any given data block, the target expert module is determined based on the matching degree between the data block and the multiple expert modules included in the target model;

[0008] Based on the target expert module, the routing vector for this data block is determined;

[0009] Based on the routing vectors of multiple data blocks and the mapping table between expert modules and devices, the target data block to be processed for each device is determined.

[0010] For any target data block to be processed on a remote device, the difference parameters of the target data block relative to the shared basic parameters are determined. Based on the difference parameters, a residual data block is generated to obtain an aggregation request packet. The aggregation request packet is sent to the remote device so that the target expert module of the remote device can perform inference based on the aggregation request packet and obtain the inference result of the target data block.

[0011] For any data block, based on the routing vector of the data block, obtain the reasoning result of each target expert module for the data block in the routing vector, and determine the target reasoning result of the data block based on the reasoning result.

[0012] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described model lightweighting methods when executing the computer program.

[0013] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described model lightweighting methods.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model lightweighting methods.

[0015] This application improves resource utilization by dividing the input sequence of the target model into multiple data blocks; for any data block, determining the target expert module based on the matching degree between the data block and the multiple expert modules included in the target model; determining the routing vector of the data block based on the target expert module; determining the target data block to be processed by each device based on the routing vectors of multiple data blocks and the mapping table between expert modules and devices; for any target data block to be processed by a remote device, determining the difference parameters of the target data block relative to the shared basic parameters; generating a residual data block based on the difference parameters to obtain an aggregation request packet; sending the aggregation request packet to the remote device so that the target expert module of the remote device can perform inference based on the aggregation request packet and obtain the inference result of the target data block; and for any data block, obtaining the inference result of each target expert module in the routing vector for that data block based on the routing vector of the data block, and determining the target inference result of the data block based on the inference result. By calculating the matching degree between the data block and the expert module and determining the routing vector of the data block, the matching efficiency of the data block is improved. By determining the difference parameters between the target data block and the shared basic parameters, residual data blocks are generated based on the difference parameters to obtain an aggregation request packet. The aggregation request packet is then sent to a remote device. This approach reduces data storage, data transmission, and computation while preserving key parameter information to the greatest extent possible, thus ensuring inference accuracy. Therefore, it solves the technical problem in related technologies where the method of achieving lightweight large models reduces inference accuracy, achieving the technical effect of ensuring inference accuracy while achieving lightweight large models. Attached Figure Description

[0016] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of a lightweight model system provided in this application embodiment;

[0018] Figure 2A flowchart illustrating a model lightweighting method provided in an embodiment of this application;

[0019] Figure 3 A flowchart illustrating yet another model lightweighting method provided in an embodiment of this application;

[0020] Figure 4 A flowchart illustrating the process of determining the routing vector of a data block, provided in an embodiment of this application;

[0021] Figure 5 A flowchart for processing data blocks provided in this application embodiment;

[0022] Figure 6 A schematic diagram illustrating the process of obtaining an aggregation request packet provided in an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0025] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0026] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] As the scale of large-scale machine learning models expands from tens of billions to trillions of parameters, the demand for hardware resources during inference processes grows exponentially. These resources include graphics processing units (GPUs), memory, computing cores, and network bandwidth. Deploying large models on a single device is insufficient to meet the demands for low-cost, low-latency deployment. To address this issue, related technologies utilize multi-device deployment of large models to meet these requirements. Large models refer to machine learning models with massive parameters and complex computational structures. They are designed to enhance expressive power and predictive performance, enabling them to handle more complex tasks and data. Large models have wide applications across various fields, including natural language processing, computer vision, speech recognition, and recommender systems.

[0028] When deploying large models across multiple devices, relevant technologies achieve lightweighting of large models through static model compression, static model parallelism, and static expert systems. Static model compression reduces model parameter size by using post-training pruning (fixed removal of redundant parameters), quantization (compressing 32-bit floating-point numbers to 16-bit or 8-bit integers), or knowledge distillation (using smaller models to learn from larger models), thus achieving lightweighting. Static model parallelism splits large models layer by layer or attention focus across different devices (e.g., multiple GPUs), with each device responsible for only a fixed module's computation, and cross-device data transfer following fixed links, achieving lightweighting of large models. Static expert systems process input sequences using fixed routing rules (e.g., allocating expert modules based on input sequence position), achieving lightweight processing of large models.

[0029] Static model compression technology achieves lightweighting of large models by pre-setting pruning and quantization methods, directly discarding a large number of parameters or reducing parameter precision. While this reduces the storage and computational load of large models, it also reduces their inference accuracy. Furthermore, the compression strategy of static model compression technology needs to be pre-set, and cannot be dynamically adjusted based on real-time hardware resource status (such as GPU utilization and memory usage) and input data characteristics (such as text complexity and image resolution) throughout the training and inference cycle. For example, when handling sudden high-complexity text tasks, because a large number of parameters have been pre-set and pruned, the large model lacks sufficient computational resources to handle complex semantic understanding, resulting in a high inference error rate. Even when hardware resources are sufficient, redundant computing units cannot be dynamically activated, leading to idle computing power.

[0030] In static model parallel technology, cross-device data transmission needs to be executed according to a fixed link, which lacks a flexible scheduling mechanism, resulting in low data transmission efficiency.

[0031] Static expert system technology does not calculate the matching degree between data blocks and expert modules, resulting in a large number of expert modules processing mismatched data blocks, low resource utilization, and reduced inference accuracy.

[0032] Taking the task of processing financial text risk control as an example, after using the lightweight method of large models in related technologies, the accuracy of large models in identifying fake risk control texts dropped from 95% to below 80%, resulting in a large number of risky transactions being missed and seriously affecting financial security.

[0033] To address the aforementioned technical problems, embodiments of this application provide a model lightweighting method and an electronic device. The model lightweighting method includes: dividing the input sequence of a target model into multiple data blocks; for any data block, determining a target expert module based on the matching degree between the data block and multiple expert modules included in the target model; determining a routing vector for the data block based on the target expert module; determining a target data block to be processed by each device based on the routing vectors of multiple data blocks and a mapping table between expert modules and devices; for any target data block to be processed by a remote device, determining a difference parameter of the target data block relative to shared basic parameters, generating a residual data block based on the difference parameter to obtain an aggregation request packet, sending the aggregation request packet to the remote device so that the target expert module of the remote device can perform inference based on the aggregation request packet to obtain the inference result of the target data block; for any data block, obtaining the inference result of each target expert module in the routing vector for the data block based on the routing vector, and determining the target inference result of the data block based on the inference result. The method provided by the above scheme determines the difference parameters of the target data block relative to the shared basic parameters, generates residual data blocks based on the difference parameters, obtains an aggregation request packet, and sends the aggregation request packet to a remote device. While reducing the amount of data storage, data transmission, and computation, it retains key parameter information to the greatest extent and ensures inference accuracy. Therefore, it can solve the technical problem that the method of achieving large model lightweighting in related technologies reduces inference accuracy, and achieves the technical effect of ensuring inference accuracy while achieving large model lightweighting.

[0034] The specific application environment architecture or specific hardware architecture on which the execution of the model lightweighting method depends is described here.

[0035] The model lightweighting method and electronic device provided in this application are suitable for achieving model lightweighting while maintaining inference accuracy. Figure 1The diagram shown is a structural schematic of the lightweight model system upon which this application is based. The lightweight model system includes a client, a local device, and a remote device. The multiple expert modules included in the target model are distributed across the local device and the remote device. The device where the entry point of the target model is located is the local device, and the device that is different from the local device is the remote device. The process involves the client inputting the input sequence of the target model into a local device, which then divides the input sequence into multiple data blocks. For any given data block, a target expert module is determined based on its matching degree with the multiple expert modules included in the target model. Based on the target expert module, a routing vector for that data block is determined. Based on the routing vectors of multiple data blocks and the mapping table between expert modules and devices, a target data block to be processed for each device is determined. For any target data block to be processed on a remote device, a difference parameter relative to the shared basic parameters is determined. Based on the difference parameter, a residual data block is generated to obtain an aggregation request packet. This aggregation request packet is sent to the remote device so that the expert modules in the remote device can perform inference based on the aggregation request packet and obtain the inference result for the target data block. For any given data block, based on the routing vector of that data block, the inference result of each target expert module in the routing vector for that data block is obtained. Based on the inference result, the target inference result for that data block is determined.

[0036] Embodiments of this application provide a model lightweighting method applied to the aforementioned local device. Figure 2 This is a flowchart illustrating the model lightweighting method provided in the embodiments of this application, as shown below. Figure 2 As shown, the process includes the following steps:

[0037] Step S201: Divide the input sequence of the target model into multiple data blocks.

[0038] The target model is a large model. The input sequence is divided into data blocks of fixed size B×S, where B is the batch size and S is the sequence length.

[0039] The size of the data block needs to be matched with the characteristics of the input sequence (such as semantic coherence and local correlation of features). Understandably, if S is too large (e.g., exceeding the semantic segment length of the input sequence), the data block may contain too much irrelevant information, leading to ambiguity in the matching degree calculation. If S is too small, it will increase the number of data blocks and the frequency of determining the corresponding target expert module, introducing additional scheduling overhead.

[0040] In this embodiment, the granularity of the input sequence is set according to the type of the input sequence (such as text, image, or voice).

[0041] Step S202: For any data block, determine the target expert module based on the matching degree between the data block and the multiple expert modules included in the target model.

[0042] Specifically, the dynamic sparse routing layer calculates the relationship between each data block and the set of expert modules. The matching degree of each expert module in the data is used to determine the target expert module that matches each data block.

[0043] Understandably, the set of expert modules is determined based on the multiple expert modules included in the target model.

[0044] It should be noted that expert modules are computational modules with specialized functions. That is, each expert module can optimize for specific types of data blocks (such as specific semantic topics or feature distributions), making the matching degree between data blocks and expert modules meaningful. For example, in Natural Language Processing (NLP) tasks, some expert modules focus on processing syntactic structures, while others focus on sentiment analysis. This differentiation is the basis for the dynamic sparse routing layer to select target expert modules.

[0045] The dynamic sparse routing layer, which calculates the matching degree, needs to be trained to learn how to measure the matching degree between data blocks and expert modules, ensuring that the output matching degree truly reflects the expert module's processing capability for the data blocks. If the dynamic sparse routing layer is not trained or is insufficiently trained, the obtained matching degree may be distorted. The dynamic sparse routing layer in this embodiment is a fully trained dynamic sparse routing layer.

[0046] Step S203: Determine the routing vector for the data block based on the target expert module.

[0047] Step S204: Based on the routing vectors of multiple data blocks and the mapping table between the expert module and the device, determine the target data block to be processed for each device.

[0048] Step S205: For any target data block to be processed by a remote device, determine the difference parameters of the target data block relative to the shared basic parameters, generate a residual data block based on the difference parameters to obtain an aggregation request packet, and send the aggregation request packet to the remote device so that the expert module in the remote device can perform inference based on the aggregation request packet to obtain the inference result of the target data block.

[0049] It should be noted that, for the target data block to be processed on the local device, the local device directly loads the target data block, determines the expert module corresponding to the target data block in the local device based on the routing vector of the target data block, and performs inference on the target data block based on the expert module corresponding to the target data block in the local device to obtain the inference result of the target data block.

[0050] Among them, the shared base parameters are the benchmark parameters used to calculate the differences in data block parameters. They serve as a reference standard in cross-device request processing, allowing the residual encoder to extract the difference parameters of data blocks relative to the shared base parameters. They do not change with the changes in individual data blocks.

[0051] The difference parameter is the core data obtained by the residual encoder after processing the data block. Specifically, it refers to the difference between each data block and the shared basic parameter, reflecting the amount of personalized parameter adjustment required for data block processing.

[0052] The function of a residual encoder is to compress the difference parameters of each data block to obtain residual data blocks. The residual data block is the final data unit generated by the residual encoder after compressing the difference parameters; it is a key data carrier for cross-device transmission, and its compression effect must meet preset requirements.

[0053] Step S206: For any data block, based on the routing vector of the data block, obtain the reasoning result of each target expert module for the data block in the routing vector, and determine the target reasoning result of the data block based on the reasoning result.

[0054] The model lightweighting method provided in this application improves resource utilization by calculating the matching degree between data blocks and expert modules to determine the routing vector of data blocks. By determining the difference parameters between the target data block and the shared basic parameters, residual data blocks are generated based on these difference parameters to obtain an aggregation request packet. This aggregation request packet is then sent to a remote device. While reducing data storage, data transmission, and computational load, this method retains key parameter information to the greatest extent possible, ensuring inference accuracy. Therefore, it solves the technical problem in related technologies where large model lightweighting methods reduce inference accuracy, achieving the technical effect of ensuring inference accuracy while achieving large model lightweighting.

[0055] The model lightweighting method provided in this application is widely applicable to tasks that rely on large models, such as natural language processing, computer vision, and voice interaction.

[0056] Embodiments of this application provide a model lightweighting method applied to local devices. Figure 3 This is a flowchart illustrating the model lightweighting method provided in the embodiments of this application, as shown below. Figure 3 As shown, the process includes the following steps:

[0057] Step S301: Divide the input sequence of the target model into multiple data blocks. See details below. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0058] Step S302: For any data block, determine the target expert module based on the matching degree between the data block and the multiple expert modules included in the target model.

[0059] Specifically, step S302 includes:

[0060] Step S3021: Based on the matching degree between the data block and the multiple expert modules included in the target model, determine multiple first expert modules whose matching degree with the data block is higher than the first matching degree threshold.

[0061] The first matching threshold is set by technical personnel. This first matching threshold must ensure that a sufficient number of valid expert modules are retained and invalid expert modules are filtered out.

[0062] Step S3022: Sort the multiple first expert modules in descending order of matching degree to obtain the sorting result.

[0063] Step S3023: Based on the sorting results, determine a preset number of first expert modules that rank highly as target expert modules.

[0064] The preset quantity is less than the quantity threshold. The quantity threshold needs to match the parallel computing capabilities of the hardware (such as the number of GPU cores and memory bandwidth) to avoid hardware resource overload due to an excessively large quantity threshold.

[0065] Figure 4 This is a flowchart illustrating the process of determining the routing vector of a data block, as provided in an embodiment of this application. Figure 4 As shown, the process includes:

[0066] The input sequence is divided into data blocks of fixed size.

[0067] The matching degree between each data block and the expert set is calculated using a dynamic sparse routing layer.

[0068] For any given data block, the target expert module for that data block is determined based on the matching degree between that data block and the expert set.

[0069] Based on the target expert module of the data block, a routing vector for the data block is generated. See steps S3021 to S3023 above for details, which will not be repeated here.

[0070] Step S303: Based on the target expert module, determine the routing vector for the data block. See details below. Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0071] Step S304: Based on the routing vectors of multiple data blocks and the mapping table between the expert module and the device, determine the target data block to be processed by each device. For details, please refer to [link to relevant documentation]. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0072] Step S305: For any target data block to be processed on a remote device, determine the difference parameters between the target data block and the shared basic parameters. Based on the difference parameters, generate a residual data block to obtain an aggregation request packet. Send the aggregation request packet to the remote device so that the expert module in the remote device can perform inference based on the aggregation request packet and obtain the inference result of the target data block. For details, please refer to [link to details]. Figure 2 Step S205 of the illustrated embodiment will not be described again here.

[0073] Figure 5 The flowchart for processing data blocks provided in the embodiments of this application is as follows: Figure 5 As shown, the process includes:

[0074] Get the routing vector of the data block.

[0075] Query the mapping table between the expert module and the device.

[0076] Identify the device where each target expert module is located in the routing vector.

[0077] Determine whether the device containing the target expert module is a remote device.

[0078] If the target expert module is located on a remote device, a cross-device request merging and residual coding process is executed, and the target expert module on the remote device processes the corresponding data block. See step S305 for details, which will not be repeated here.

[0079] If the target expert module is located on a local device, the calculation process of the target expert module on the local device is executed asynchronously and in parallel, and the target expert module on the local device processes the corresponding data blocks. See the relevant description of step S205 for details, which will not be repeated here.

[0080] Step S306: For any data block, based on the routing vector of that data block, obtain the reasoning result of each target expert module for that data block in the routing vector, and determine the target reasoning result for that data block based on the reasoning result. See details below. Figure 2 Step S206 of the illustrated embodiment will not be described again here.

[0081] The model lightweighting method provided in this application calculates the matching degree between data blocks and expert modules in real time through a dynamic sparse routing layer, determines the target expert module with a high matching degree with the data block, and uses the target expert module with a high matching degree to process the data block, thereby improving resource utilization.

[0082] In some optional implementations, prior to step S304, the model lightweighting method further includes:

[0083] Step a1: If the resource utilization rate of the local device where the entry point of the target model is located is greater than the first resource utilization threshold, then based on the first formula, the preset quantity is reduced, and the routing vector of the data block is updated based on the reduced preset quantity. The first formula is:

[0084]

[0085] in, To reduce the preset quantity, To reduce the preset quantity, This is the resource scaling factor. The resource scaling factor is set by technical personnel and is a value less than 1.

[0086] Step a2: If the resource utilization rate of the local device where the entry point of the target model is located is less than the second resource utilization rate threshold, then increase the preset quantity, and update the routing vector of the data block based on the increased preset quantity.

[0087] The first resource utilization threshold, i.e., the high load threshold, is greater than the second resource utilization threshold, i.e., the low load threshold. If the resource utilization of the local device is less than the second resource utilization threshold, the preset quantity is increased to improve inference accuracy.

[0088] Understandably, based on the updated routing vectors of multiple data blocks and the mapping table between expert modules and devices, the target data block to be processed by each device is determined, ensuring that the expert module scheduling is adapted to the real-time resource status.

[0089] The model lightweighting method provided in this application dynamically adjusts the number of target expert modules by combining hardware resource utilization with a preset number. When dealing with sudden high-complexity tasks, it can ensure semantic understanding capabilities by increasing the number of effective activated expert modules, avoiding a surge in inference error rates. When hardware resources are sufficient, it can also dynamically utilize redundant computing units to reduce idle computing power. This ensures that expert module scheduling is adapted to real-time resource status.

[0090] In some optional implementations, prior to step S304, the model lightweighting method further includes:

[0091] Step b1: For any target expert module, determine whether the current load rate of the target expert module is less than the preset load rate threshold, and perform integrity verification on the target expert module.

[0092] The preset load rate threshold is set by technical personnel.

[0093] Step b2: If the current load rate of the target expert module is not less than the preset load rate threshold or the integrity verification of the target expert module fails, then the candidate expert module of the target expert module is determined based on the matching degree between the data block and the multiple expert modules included in the target model.

[0094] If the current load rate of the target expert module is not less than the preset load rate threshold or the integrity verification of the target expert module fails, then the target expert module is determined to be an invalid expert module. Based on the matching degree between the data block and the multiple expert modules included in the target model, the expert module with the highest matching degree with the data block is determined from the other expert modules besides the identified target expert module as the candidate expert module of the target expert module.

[0095] Step b3: Update the routing vector for the data block based on the candidate expert module.

[0096] Specifically, the routing vector of the data block is updated by replacing the target expert module in the routing vector with the candidate expert module of the target expert module.

[0097] The model lightweighting method provided in this application determines the routing vector more accurately by judging whether the current load rate of the target expert module is less than a preset load rate threshold and performing integrity verification on the target expert module.

[0098] In some optional implementations, prior to step S304, the model lightweighting method further includes:

[0099] Step c1: Obtain all historical routing vectors within the preset time period.

[0100] The preset time period is set by technical personnel and no specific restrictions are imposed here.

[0101] Step c2: Based on all historical routing vectors, determine the co-activation probability of any two expert modules being included in the same data block of routing vectors.

[0102] Among them, an expert collaboration matrix is ​​constructed based on all historical routing vectors. The expert collaboration matrix is ​​a statistical analysis of all historical routing vectors. The elements in the expert collaboration matrix... Representative Expert Module and The probability of being simultaneously activated by the same input data block (i.e., the co-activation probability of any two expert modules being included in the routing vector of the same data block). When ( When the preset co-activation probability threshold is used, the explanation is as follows: and There is a strong collaborative relationship – if the two are currently on different devices, they are likely to trigger cross-device requests simultaneously in subsequent steps, increasing communication overhead; if they are bound and scheduled to the same device, the two cross-device requests can be merged into a single local device request.

[0103] It should be noted that the expert collaboration matrix is ​​updated periodically based on the latest historical routing vectors to ensure that the expert collaboration matrix reflects the latest collaboration patterns.

[0104] Step c3: If the collaborative activation probability of two expert modules included in the routing vector of the same data block is greater than the preset collaborative activation probability threshold, then the device to which the two expert modules belong is determined based on the mapping table between expert modules and devices.

[0105] Step c4: If the two expert modules belong to different devices, then migrate the expert module with the lower load to the device where the expert module with the higher load belongs, and update the mapping table between expert modules and devices.

[0106] By migrating the expert module with lower load from the two expert modules to the device where the expert module with higher load is located, binding scheduling is completed, reducing frequent cross-device communication caused by the cross-device distribution of highly collaborative expert modules.

[0107] The model lightweighting method provided in this application reduces communication frequency and lowers cross-device communication costs by migrating the expert module with lower load to the device where the expert module with higher load resides if the collaborative activation probability of two expert modules included in the same data block of the routing vector is greater than a preset collaborative activation probability threshold, and the two expert modules belong to different devices.

[0108] In some optional implementations, prior to step S304, the model lightweighting method further includes:

[0109] Step d1: Based on the matching degree between the data block and the multiple expert modules included in the target model, identify multiple second expert modules whose matching degree with the data block is higher than the second matching degree threshold.

[0110] The second matching threshold is set by technicians. If the matching degree between an expert module and the data block is higher than the second matching threshold, it means that the expert module is very likely to be activated by the data block. That is, the expert module is very likely to be the target expert module of the data block. If the expert module is located on a remote device at this time, the expert module will be migrated to the cache of the local device in advance.

[0111] Step d2: Based on the mapping table between expert modules and devices, determine the device where the second expert module is located.

[0112] Step d3: If the device where the second expert module is located is a remote device, then migrate the second expert module to the cache of the local device and update the mapping table between the expert module and the device.

[0113] It should be noted that migrating the second expert module to the local device's cache includes: utilizing the local device's current idle bandwidth, without blocking the aforementioned route vector adjustment steps, and preloading the second expert module into the local device's cache, such as the GPU's High-Bandwidth Memory (HBM) cache.

[0114] Understandably, migrating the second expert module to the local device's cache adopts an asynchronous, partially blocking mode. This means the migration of the second expert module is executed in parallel with the aforementioned routing vector adjustment steps, avoiding additional process latency. Simultaneously, only expert modules with high matching scores are migrated to ensure cost-effectiveness of resource investment and prevent low-matching expert modules from occupying the local cache.

[0115] The model lightweighting method provided in this application reduces cross-device communication costs by migrating the second expert module to the cache of the local device in advance. This allows more than 90% of the frequently accessed expert modules to be called directly on the local device.

[0116] In some alternative implementations, step d3 above includes:

[0117] Step d31: If the remaining space in the local device's cache is not less than a preset remaining space threshold, migrate the second expert module to the local device's cache.

[0118] The preset remaining space threshold is set by technical personnel. The cache space of the local device is limited by the size of the local device's memory.

[0119] Step d32: If the remaining space of the local device's cache is less than a preset remaining space threshold, obtain the elimination score of each expert module in the local device's cache. Based on the elimination score, migrate the expert module with the lowest elimination score to its original device and migrate the second expert module to the local device's cache.

[0120] Specifically, among the most recent preset number of accesses on the local device, the expert module with the lowest access count and the earliest recent access timestamp in the local device's cache receives a lower elimination score. This ensures that the expert module with the lowest access count and the earliest recent access timestamp among the most recent preset number of accesses is prioritized for elimination, ensuring that the expert module most likely to be used by subsequent data blocks is always retained in the cache. The preset number of accesses is set by technical personnel and is not specifically limited here.

[0121] Understandably, after migrating the expert module with the lowest elimination score to its original device and migrating the second-highest expert module to the local device's cache, the mapping table between expert modules and devices is updated.

[0122] It should be noted that, based on the routing vectors of multiple data blocks and the mapping table between the expert module and the device, the target data block to be processed by each device is determined, including:

[0123] Based on the routing vectors of multiple data blocks and the mapping table between expert modules and devices, the set of remote expert modules and the set of local expert modules located on remote devices are determined.

[0124] For any remote expert module in the set of remote expert modules, determine the set of data blocks to be processed corresponding to that remote expert module.

[0125] The set of data blocks to be processed corresponding to the remote expert module is filtered for validity to obtain the target data block set corresponding to the remote expert module. Specifically, for any data block to be processed in the set of data blocks to be processed corresponding to the remote expert module, if the matching degree between the data block to be processed and the remote expert module is lower than a third matching degree threshold, the data block to be processed is deleted from the set of data blocks to be processed corresponding to the remote expert module to avoid invalid communication. The third matching degree threshold is set by technical personnel.

[0126] Based on the set of target data blocks corresponding to each remote expert module, the target data blocks to be processed for each remote device are determined, that is, a mapping table between remote devices and data blocks is generated.

[0127] For any local expert module in the local expert module set, determine the set of data blocks to be processed corresponding to that local expert module. Based on the set of data blocks to be processed corresponding to each local expert module, determine the target data block to be processed by the local device, that is, generate a mapping table between the local device and the data block.

[0128] The lightweight model method provided in this application avoids the unrestricted occupation of local device cache by migrated expert modules through cache management, thus preventing local device resource overload. By dynamically phasing out low-value expert modules, it ensures that high-value expert modules can stably reside in the cache without affecting the normal computation of the local device.

[0129] In some alternative implementations, step d32 above includes:

[0130] Step d321: Obtain the access timestamp and access count of each expert module in the cache of the local device in the most recent preset number of accesses on the local device.

[0131] Specifically, an access counter is maintained for each expert module in the local device's cache, recording the access timestamp and access count of each expert module in the local device's cache within the most recent preset number of accesses.

[0132] Step d322: Based on the access timestamp and access count of each expert module in the local device's cache during the most recent preset number of accesses on the local device, determine the elimination score of each expert module in the local device's cache.

[0133] Specifically, based on the access timestamp of each expert module in the cache of the local device within the most recent preset number of accesses on the local device, the most recent access timestamp of each expert module in the cache of the local device is determined.

[0134] Obtain the weights of the most recent access timestamp and the access count. The weights of the most recent access timestamp and the access count are set by technical personnel.

[0135] Based on the most recent preset number of accesses on the local device, the most recent access timestamp and access count of each expert module in the local device's cache, the weight of the most recent access timestamp and the weight of the access count are used to determine the elimination score of each expert module in the local device's cache.

[0136] Specifically, based on a preset correspondence table between access timestamps and first scores, the target first score corresponding to the most recent access timestamp of each expert module in the local device's cache is determined.

[0137] Based on a pre-defined table of correspondence between access counts and second scores, the target second score corresponding to the access count of each expert module in the local device's cache is determined.

[0138] The elimination score for each expert module in the local device's cache can be determined using the following formula:

[0139] Elimination score = Target first score corresponding to the most recent access timestamp of the expert module × Weight of the most recent access timestamp + Target second score corresponding to the number of accesses to the expert module × Weight of the number of accesses.

[0140] In some optional implementations, step S305 above includes:

[0141] Step e1: For any target data block to be processed by the remote device, obtain the cosine similarity between the difference parameter of the target data block and the difference parameter of each target data block in other target data blocks to be processed by the remote device.

[0142] Step e2: If at least one of the difference parameters of the first target data block and the difference parameter of the target data block have a cosine similarity greater than a preset similarity threshold, then a first dictionary entry in the shared dictionary is generated based on the difference parameters of the target data block and the difference parameters of the first target data block.

[0143] The preset similarity threshold is set by technical personnel; for example, the preset similarity threshold could be 0.85. The difference parameters of the target data block and the first target data block are clustered to generate a first dictionary entry in a shared dictionary. It can be understood that the difference parameter of the target data block can be represented as the index of the first dictionary entry plus the difference between the difference parameter of the target data block and the first dictionary entry. The difference parameter of the first target data block can also be represented as the index of the first dictionary entry plus the difference between the difference parameter of the first target data block and the first dictionary entry, thereby improving the compression rate of the residual data block to 65%.

[0144] Step e3: Based on the difference parameters of the target data block and the difference parameters of the first target data block, determine the difference sparse matrix of the target data block and the difference sparse matrix of the first target data block.

[0145] Specifically, for any element in the difference parameters of the target data block, if the value of that element is less than a preset value, then that element is set to 0, thus obtaining the difference sparse matrix of the target data block. The preset value is set by technical personnel and is not specifically limited here.

[0146] The method for obtaining the difference sparse matrix of the first target data block is similar to that for obtaining the difference sparse matrix of the target data block, and will not be repeated here.

[0147] Step e4: Based on the difference sparse matrix of the target data block, logarithmic processing is performed on the non-zero values ​​in the difference sparse matrix of the target data block to obtain the parameter difference matrix of the target data block.

[0148] Specifically, the non-zero values ​​of the differential sparse matrix and their position indices are stored using a Compressed Sparse Row (CSR) structure.

[0149] The parameter difference matrix of the target data block is obtained by performing a logarithmic quantization on each non-zero value in the difference sparse matrix. Specifically, the logarithmic quantization is performed on each non-zero value using the following formula to obtain the integer code corresponding to each non-zero value:

[0150]

[0151] in, It represents the integer code corresponding to the non-zero value in the differential sparse matrix. This is the floor function. It is a very small positive number and is set by technical personnel. The scaling factor controls the scale density of the log field: each additional quantization scale is equivalent to multiplying the real number field by 2^(1 / b). The larger b is, the denser the scale becomes, resulting in more codewords and higher precision but larger size; the smaller b is, the opposite is true. In practice, this value is designed in conjunction with the bit width of the data block. In the embodiments of this application, this value is set by the technicians. represents the non-zero values ​​in the differential sparse matrix.

[0152] Step e5: Based on the difference sparse matrix of the first target data block, logarithmic processing is performed on the non-zero values ​​in the difference sparse matrix of the first target data block to obtain the parameter difference matrix of the first target data block.

[0153] The method for obtaining the parameter difference matrix of the target data block is similar and will not be described in detail here.

[0154] Step e6: Based on the index of the first dictionary entry and the parameter difference matrix of the target data block, determine the residual data block corresponding to the target data block.

[0155] Understandably, the residual encoder compresses the index of the first dictionary entry and the parameter difference matrix of the target data block to generate the residual data block corresponding to the target data block. It should be noted that the compression ratio of the residual encoder is greater than a preset compression ratio threshold, determined based on the number of residual quantization bits. The preset compression ratio threshold is a minimum standard set for the compression effect of the residual data block; that is, the ratio of the compressed volume of the residual data block to the uncompressed volume must be greater than the preset compression ratio threshold to ensure the data block transmission rate across devices.

[0156] Specifically, the structure of the residual data block is as follows:

[0157] Header metadata: includes expert module identifier, shared basic parameter hash value, and residual quantization bit count. ;in, , This is the minimum value for the preset residual quantization bits. This is the maximum value of the preset residual quantization bits;

[0158] Parameter difference matrix;

[0159] Shared dictionary index: Points to the common feature vector in the shared dictionary generated by clustering, with an index width of [number missing]. In this embodiment, the shared dictionary index is the index of the first dictionary entry.

[0160] Step e7: Based on the index of the first dictionary entry and the parameter difference matrix of the first target data block, determine the residual data block corresponding to the first target data block.

[0161] Step e8: Obtain the target data block to be processed by the remote device within the preset time window, and generate an aggregation request packet based on the residual data block corresponding to the target data block to be processed by the remote device within the preset time window.

[0162] The time window refers to the time interval used to aggregate multiple residual data blocks sent to the same remote device during the cross-device residual data block merging process, such as 3 seconds.

[0163] An aggregated request packet is a single request packet formed by integrating multiple residual data blocks sent to the same remote device within a preset time window according to specific rules.

[0164] It should be noted that if no other target data block has a cosine similarity greater than a preset similarity threshold between its difference parameter and that of the target data block, then based on the difference parameter of the target data block, a second dictionary entry is generated in the shared dictionary; based on the difference parameter of the target data block, the difference sparse matrix of the target data block is determined; based on the difference sparse matrix of the target data block, the non-zero values ​​in the difference sparse matrix of the target data block are logarithmically quantized to obtain the parameter difference matrix of the target data block; based on the index of the second dictionary entry and the parameter difference matrix of the target data block, the residual data block corresponding to the target data block is determined; the target data block to be processed by the remote device within a preset time window is obtained, and based on the residual data block corresponding to the target data block to be processed by the remote device within the preset time window, an aggregation request packet is generated.

[0165] It should be further explained that after receiving the aggregation request packet, the remote device sends the residual data block to the corresponding expert module according to the expert module identifier in the residual data block in the aggregation request packet. Specifically, when sending the residual data block to the corresponding expert module, if the matching degree between the expert module and the corresponding data block is greater than the fourth matching degree threshold, a Quality of Service (QoS) channel is allocated to the residual data block to ensure that the latency from the residual data block to the expert module is less than a preset latency threshold.

[0166] After receiving the corresponding residual data block, the expert module in the remote device needs to use TensorCore to recover the data block corresponding to the residual data block. Specifically, the data block corresponding to the residual data block is recovered using the following formula:

[0167]

[0168] in, This is the data block corresponding to the residual data block. To share basic parameters, For parameter operators, This refers to the inverse transformation of the quantizer. It is the integer code corresponding to the non-zero value in the difference sparse matrix in the residual data block.

[0169] It is understandable that by performing an inverse transformation of the quantizer on the integer code corresponding to each non-zero value in the difference sparse matrix of the residual data block, each non-zero value in the difference sparse matrix of the residual data block can be recovered, thereby recovering the difference sparse matrix of the residual data block. Using the difference sparse matrix of the residual data block and the shared basic parameters, the data block corresponding to the residual data block can be recovered.

[0170] It should be noted that the expert module in the remote device loads the corresponding residual data block using a zero-copy method and reconstructs the complete data block based on shared fundamental parameters. This zero-copy method can be Direct Memory Access (DMA). The zero-copy method avoids the overhead of multiple copying operations between host memory and device memory in traditional data transmission, further reducing data loading latency.

[0171] The expert module in the remote device loads the recovered data block into the stream multiprocessor register and starts expert module inference, that is, it uses the expert module to infer the recovered data block and obtain the inference result of the recovered data block.

[0172] It should be further noted that the steps of reasoning about the target data block to be processed on the local device by the expert module are executed in parallel with the steps of sending the aggregation request packet to the remote device.

[0173] Use TensorCore to recover data blocks corresponding to multiple residual data blocks in parallel.

[0174] The recovered data blocks are loaded in parallel into the streaming multiprocessor registers, and expert module inference is initiated.

[0175] By operating in parallel across these three stages, the latency of cross-device communication is minimized.

[0176] Figure 6 This is a schematic diagram of the process for obtaining the aggregation request packet provided in an embodiment of this application, such as... Figure 6 As shown, the process includes:

[0177] Obtain the set of target data blocks to be processed from the remote device.

[0178] Obtain the cosine similarity between the difference parameters of the target data blocks.

[0179] If the cosine similarity between the difference parameters of multiple target data blocks is greater than a preset similarity threshold, then a dictionary entry in a shared dictionary is generated based on the difference parameters of the multiple target data blocks.

[0180] A difference sparse matrix is ​​generated based on the difference parameters of the target data block.

[0181] Generate a parameter difference matrix based on the difference sparse matrix.

[0182] Residual data blocks are generated based on the parameter difference matrix and dictionary entry index.

[0183] Generate an aggregate request packet based on the residual data block.

[0184] The lightweight model method provided in this application improves the problem of a sharp drop in inference accuracy caused by simply discarding parameters in related technologies. By transmitting only the difference information between the data block and the shared basic parameters through residual coding, the key parameter information is preserved to the greatest extent while reducing the amount of data storage, data transmission, and computation. This avoids the loss of accuracy caused by excessive parameter simplification. In accuracy-sensitive tasks such as financial text risk control, it effectively maintains a high recognition accuracy rate to reduce risk underreporting.

[0185] In some alternative implementations, the above model lightweighting method further includes:

[0186] Step f1: If the bandwidth fluctuation of the local device exceeds a preset fluctuation threshold, adjust the preset time window based on the second formula; the second formula is:

[0187]

[0188] in, For preset time windows, and For network characteristic calibration coefficients, The standard deviation of bandwidth fluctuation for local devices. This represents the average network latency of the local device.

[0189] in, Used in the second formula to adjust The weight of the impact on the preset time window.

[0190] The settings need to be matched to network fluctuation sensitivity: if network bandwidth fluctuates frequently, the sensitivity should be increased appropriately. This allows the preset time window to adjust more sensitively to bandwidth fluctuations, avoiding request backlogs caused by sudden drops in network bandwidth; if network bandwidth is stable, it can reduce... This reduces the frequency of changes in the preset time window and reduces the additional overhead of request merging logic.

[0191] As the base offset for the preset time window, it ensures that the preset time window is always within a reasonable range, avoiding [further issues]. If the value is too small, the preset time window will be lower than the minimum value.

[0192] The settings need to ensure a balance between request merging efficiency and latency: if the task is latency-sensitive, It should be set to a value close to the minimum value of the preset time window to avoid... An excessively large value can lead to excessively long request waiting times. If the task prioritizes throughput (such as offline batch text processing), the value can be increased appropriately. This allows more requests to be combined to reduce communication frequency and improve overall efficiency.

[0193] The preset fluctuation amplitude threshold is the critical condition that triggers the reset of the preset time window. When the real-time bandwidth fluctuation amplitude exceeds this value, the calculation is recalculated. This ensures that the preset time window adapts to sudden network changes.

[0194] First, analyze the historical bandwidth fluctuation range of the local device's network environment. Then, set the preset fluctuation threshold to a value slightly higher than the maximum value in the historical bandwidth fluctuation range. This will prevent frequent resets due to minor fluctuations and enable timely responses to bandwidth changes.

[0195] The lightweight model method provided in this application ensures the merging efficiency of data blocks by adjusting a preset window when the bandwidth fluctuation of the local device exceeds a preset fluctuation threshold.

[0196] In some alternative implementations, step S306 above includes:

[0197] Step g1: For any target expert module in the routing vector of the data block, obtain the gradient compensation term of that target expert module.

[0198] The formula for calculating the gradient compensation term is as follows:

[0199]

[0200] in, This is the gradient compensation value. To share basic parameters, Let be the loss function for the target expert module. The gradient compensation coefficients of the target expert module are... For partial derivatives, in this embodiment, This represents the gradient of the loss function with respect to the shared basic parameters.

[0201] Step g2: Use the gradient compensation term of the target expert module to compensate the difference parameters of the data block to obtain the target difference parameters after compensation by the target expert module.

[0202] Understandably, based on the gradient compensation term of the target expert module, the gradient compensation value of the target expert module is determined, and based on the gradient compensation value of the target expert module, the difference parameters of the data block are compensated. Specifically, the gradient compensation value of the target expert module is added to the difference parameters of the data block to obtain the target difference parameters compensated by the target expert module.

[0203] Understandably, each target expert module in the routing vector of this data block will obtain the corresponding compensated target difference parameters.

[0204] Step g3: Obtain the norm of the target difference parameters after compensation by multiple target expert modules.

[0205] Step g4: Determine the weighting coefficient of each target expert module based on the norm of the target difference parameters after compensation by multiple target expert modules.

[0206] If the norm of the target difference parameter compensated by any target expert module is less than a preset norm threshold, then the target difference parameter compensated by the target expert module is determined as a shared basic parameter, and the norm of the target difference parameter compensated by the target expert module is determined as a shared basic parameter.

[0207] Specifically, the norms of the target difference parameters compensated by multiple target expert modules are summed to obtain the total norm value.

[0208] For any target difference parameter after compensation by the target expert module, the weighting coefficient of the target expert module is obtained by dividing the norm of the target difference parameter after compensation by the target expert module by the total norm value.

[0209] It should be noted that after obtaining the norms of the target difference parameters compensated by multiple target expert modules, the multiple compensated target difference parameters are sorted in descending order of norm. The compensated target difference parameters with higher norms are assigned higher bit widths to ensure reconstruction accuracy, while the other compensated target difference parameters are assigned lower bit widths. The specific bit widths assigned are set by the technical personnel.

[0210] Step g5: Based on the weighting coefficients of each target expert module, the reasoning results of each target expert module for the data block, and the target difference parameters compensated by each target expert module, determine the target reasoning result for the data block.

[0211] The model lightweighting method provided in this application suppresses the accumulation of quantization error by injecting gradient compensation terms, thereby balancing lightweighting and inference accuracy in the obtained target inference result.

[0212] In some alternative implementations, step g5 above includes:

[0213] Step g51: Based on the weighting coefficients of each target expert module, the inference results of each target expert module for the data block, and the target difference parameters compensated by each target expert module, determine the corrected inference results of each target expert module for the data block.

[0214] Specifically, for any target expert module, the output correction mapping function is used to map the target difference parameters compensated by the target expert module from the parameter domain to the output domain, generating a first correction amount of the same dimension as the inference result of the target expert module for the data block.

[0215] The target correction is obtained by multiplying the first correction amount by the weighting coefficient of the target expert module.

[0216] The reasoning result of the target expert module for this data block is added to the target correction amount to obtain the corrected reasoning result of the target expert module for this data block.

[0217] Step g52: Based on the corrected inference result of each target expert module for the data block and the matching degree between the data block and each target expert module, determine the target inference result of the data block.

[0218] Specifically, for any target expert module, the corrected inference result of the target expert module for the data block and the matching degree between the data block and the target expert module are multiplied to obtain the first inference result of the target expert module.

[0219] The target inference result of the data block is obtained by summing the first inference results of multiple target expert modules.

[0220] In some alternative implementations, the above model lightweighting method further includes:

[0221] If, based on the routing vector of the data block, the reasoning result of at least one second target expert module in the routing vector for that data block fails to be obtained, then the step of obtaining the reasoning result of the second target expert data block in the routing vector for that data block is re-executed.

[0222] If the number of times the step of re-executing the step of obtaining the reasoning result of the second target expert data block in the routing vector exceeds a preset threshold, an alarm will be issued.

[0223] The preset threshold for the number of attempts is set by technical personnel and is not specifically limited here.

[0224] The model lightweighting method provided in this application enhances the robustness of the system and its tolerance to transient errors by re-executing the step of obtaining the inference result of the second target expert data block in the routing vector for the data block when the inference result of at least one second target expert module in the routing vector for the data block fails.

[0225] The model lightweighting method provided in this application significantly reduces the proportion of cross-device communication latency in the overall inference latency. In traditional solutions, cross-device communication latency often becomes the main component of inference time, while this application can greatly reduce this proportion, so that the communication link no longer dominates the total inference time, and instead the computation link becomes the core determinant of inference efficiency, fundamentally optimizing the inference time allocation structure in multi-device collaborative scenarios.

[0226] The lightweight model method provided in this application can be widely applied to mobile network edge computing scenarios, extending from fixed networks to mobile networks. The change process is as follows: In mobile networks, the fixed mapping table between expert modules and devices in this application is adjusted to a dynamic device mapping table to adapt to the dynamic topology between base stations, edge nodes, and terminal devices. For example, when a user terminal switches between base stations, the location of the edge node where the expert module is located is updated in real time, and the input sequence blocks are changed to "variable-sized data blocks" adapted to the low bandwidth characteristics of mobile networks. The length S is adjusted according to the real-time signal strength. Specifically, the system judges the current network quality through the Reference Signal Received Power (RSRP) and Signal to Interference plus Noise Ratio (SINR) reported by the terminal in real time. When the signal strength is strong, the sequence length is set to a larger value to reduce the number of blocks and scheduling overhead; when the signal is weak, the column length is automatically shortened to reduce the transmission volume of a single data block and avoid data block packet loss due to insufficient bandwidth. At the same time, the batch size B is linked with the terminal computing power to balance computing efficiency and terminal power consumption.

[0227] Based on steps c1 to c4 and d1 to d3, a new step is added: "expert migration prediction based on user movement trajectory". This step, combined with GPS positioning, pre-migrates high-probability expert modules from the core network to the edge base station where the user will be connected, and requests merging time windows across devices. Change to dynamic adjustment based on mobile network latency fluctuations, for example, increasing the latency when the signal is weak. To reduce the number of requests, the residual data block compression rate is increased to a higher compression rate to adapt to the limited bandwidth of mobile networks, ultimately enabling lightweight inference of large models on mobile terminals in low-bandwidth, high-mobility scenarios. This includes applications such as terminal-side artificial intelligence (AI) voice assistants and real-time image recognition.

[0228] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0229] Embodiments of this application also provide an electronic device, such as... Figure 7 As shown, it includes a processor 701 and a memory 702, in which a computer program is stored. The processor 701 is configured to run the computer program to perform the steps in any of the above-described model lightweighting method embodiments.

[0230] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model lightweighting method embodiments at runtime.

[0231] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0232] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model lightweighting method embodiments.

[0233] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model lightweighting method embodiments.

[0234] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0235] The above provides a detailed description of a model lightweighting method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model lightweight method, characterized in that, The method comprises the following steps: dividing an input sequence of a target model into a plurality of data blocks; for any data block, determining a target expert module based on the matching degree of the data block and a plurality of expert modules included in the target model; determining a routing vector of the data block based on the target expert module; based on the routing vectors of a plurality of data blocks and a mapping table of expert modules and devices, determining target data blocks to be processed by each device, the devices including local devices and remote devices; for any target data block to be processed by a remote device, determining a difference parameter of the target data block relative to a shared base parameter, generating a residual data block based on the difference parameter to obtain an aggregation request packet, and sending the aggregation request packet to the remote device to enable an expert module in the remote device to perform inference based on the aggregation request packet to obtain an inference result of the target data block; for any data block, obtaining the inference result of each target expert module for the data block based on the routing vector of the data block, and determining a target inference result of the data block based on the inference result; the step of determining the target inference result of the data block based on the inference result comprises: for any target expert module in the routing vector of the data block, obtaining a gradient compensation term of the target expert module; compensating the difference parameter of the data block using the gradient compensation term of the target expert module to obtain a target difference parameter of the target expert module after compensation; obtaining the norm of a plurality of target difference parameters of the target expert modules after compensation; determining a weighting coefficient of each target expert module based on the norm of a plurality of target difference parameters of the target expert modules after compensation; determining the target inference result of the data block based on the weighting coefficient of each target expert module, the inference result of each target expert module for the data block, and the target difference parameter of each target expert module after compensation; the step of determining the target inference result of the data block based on the weighting coefficient of each target expert module, the inference result of each target expert module for the data block, and the target difference parameter of each target expert module after compensation comprises: determining the corrected inference result of each target expert module for the data block based on the weighting coefficient of each target expert module, the inference result of each target expert module for the data block, and the target difference parameter of each target expert module after compensation; determining the target inference result of the data block based on the corrected inference result of each target expert module for the data block and the matching degree of the data block and each target expert module.

2. The method of claim 1, wherein, the step of determining the target expert module based on the matching degree of the data block and a plurality of expert modules included in the target model comprises: determining a plurality of first expert modules with a matching degree higher than a first matching degree threshold based on the matching degree of the data block and a plurality of expert modules included in the target model; sorting a plurality of first expert modules in descending order of matching degree to obtain a sorting result; determining a preset number of first expert modules in the front of the sorting as the target expert module based on the sorting result.

3. The method of claim 2, wherein, Before determining the target data block to be processed by each device based on the routing vectors of the plurality of data blocks and the mapping table of the expert modules and devices, the method further comprises: if the resource utilization of the local device where the entry of the target model is located is greater than a first resource utilization threshold, then based on a first formula, the preset number is reduced to update the routing vector of the data block based on the reduced preset number; the first formula is: wherein, is the reduced preset number, is the preset number before reduction, is a resource scaling factor; if the resource utilization of the local device where the entry of the target model is located is less than a second resource utilization threshold, then the preset number is increased to update the routing vector of the data block based on the increased preset number; the first resource utilization threshold is greater than the second resource utilization threshold.

4. The method of claim 1, wherein, Before determining the target data block to be processed by each device based on the routing vectors of the plurality of data blocks and the mapping table of the expert modules and devices, the method further comprises: for any target expert module, it is judged whether the current load rate of the target expert module is less than a preset load rate threshold, and the target expert module is verified for integrity; if the current load rate of the target expert module is not less than the preset load rate threshold or the integrity verification of the target expert module fails, then based on the matching degree of the data block and the plurality of expert modules included in the target model, the candidate expert module of the target expert module is determined; based on the candidate expert module, the routing vector of the data block is updated.

5. The method of claim 1, wherein, Before determining the target data block to be processed by each device based on the routing vectors of the plurality of data blocks and the mapping table of the expert modules and devices, the method further comprises: all historical routing vectors in a past preset time period are obtained; based on all the historical routing vectors, the cooperative activation probability of any two expert modules being included in the routing vector of the same data block is determined; if the cooperative activation probability of two expert modules being included in the routing vector of the same data block is greater than a preset cooperative activation probability threshold, then based on the mapping table of the expert modules and devices, the devices to which the two expert modules belong are determined; if the devices to which the two expert modules belong are different devices, then the expert module with lower load among the two expert modules is migrated to the device where the expert module with higher load among the two expert modules is located, and the mapping table of the expert modules and devices is updated.

6. The method of claim 1, wherein, Before determining the target data block to be processed by each device based on the routing vectors of the plurality of data blocks and the mapping table of the expert modules and devices, the method further comprises: based on the matching degree of the data block and the plurality of expert modules included in the target model, a plurality of second expert modules with a matching degree higher than a second matching degree threshold are determined; based on the mapping table of the expert modules and devices, the devices where the second expert modules are located are determined; if the devices where the second expert modules are located are remote devices, then the second expert modules are migrated to the cache of the local device, and the mapping table of the expert modules and devices is updated.

7. The method of claim 1, wherein, The method for determining the difference parameter of the target data block to be processed by any remote device relative to the shared basic parameter, generating a residual data block based on the difference parameter to obtain an aggregation request packet, comprises: For any target data block to be processed by the remote device, obtain the cosine similarity between the difference parameter of the target data block and the difference parameter of each of the other target data blocks to be processed by the remote device; If the cosine similarity between the difference parameter of the target data block and the difference parameter of at least one first target data block among the other target data blocks is greater than a preset similarity threshold, generate a first dictionary entry in the shared dictionary based on the difference parameter of the target data block and the difference parameter of the first target data block; Determine the difference sparse matrix of the target data block and the difference sparse matrix of the first target data block based on the difference parameter of the target data block and the difference parameter of the first target data block; Perform quantization processing on the non-zero values in the difference sparse matrix of the target data block based on the difference sparse matrix of the target data block, to obtain a parameter difference matrix of the target data block; Perform quantization processing on the non-zero values in the difference sparse matrix of the first target data block based on the difference sparse matrix of the first target data block, to obtain a parameter difference matrix of the first target data block; Determine the residual data block corresponding to the target data block based on the index of the first dictionary entry and the parameter difference matrix of the target data block; Determine the residual data block corresponding to the first target data block based on the index of the first dictionary entry and the parameter difference matrix of the first target data block; Obtain the target data blocks to be processed by the remote device within a preset time window, and generate the aggregation request packet based on the residual data blocks corresponding to the target data blocks to be processed by the remote device within the preset time window.

8. An electronic device, comprising: Comprise: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the model lightweight method according to any one of claims 1 to 7.

Images