Data processing method and device, electronic equipment, storage medium and program product
By using the shared weight of the first memory and the target low-rank matrix of the second memory in the mixed expert model, the problem of low-quality data processing efficiency caused by the large model parameter volume is solved, and a significant reduction in video memory footprint is achieved. Improved processing efficiency.
Patent Information
- Application Number
- CN202410987881.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-19
AI Technical Summary
Due to the large number of parameters, the hybrid expert model occupies a large amount of GPU video memory when deployed on multiple GPUs in distributed manner, resulting in low data processing efficiency.
The target expert model is constructed by sharing weights in the first memory and storing the pre-decomposed target low-rank matrix in the second memory, reducing memory footprint and improving data processing efficiency.
It significantly reduces the memory usage, improves data processing efficiency, and does not require model compression or retraining, and is suitable for any MOE large model.
Smart Images

Figure CN118981385A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of data processing technology, and specifically relates to a data processing method, device, electronic device, storage medium and program product. Background Art
[0002] Large models have quickly become a hot application in the field of artificial intelligence due to their excellent language understanding, multi-round dialogue and logical reasoning capabilities. Mixture of Experts (MOE) is a method of deep learning models. This method generally trains or integrates multiple models as multiple expert models to improve the overall effect of the model by integrating the results of multiple expert models.
[0003] Since the large hybrid expert model is composed of multiple expert models, its parameters are often very large, and multiple graphics processing units (GPUs) are required for distributed deployment and reasoning, which takes up a large amount of GPU video memory and leads to low data processing efficiency. Summary of the invention
[0004] The embodiments of the present application provide a data processing method, device, electronic device, storage medium and program product, which can solve the problem of low data processing efficiency.
[0005] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising: determining a target expert model for processing data to be processed; constructing the target expert model through shared weights of a first memory and a target low-rank matrix of a second memory, wherein the target low-rank matrix is obtained by pre-decomposing the target expert model; and processing the data to be processed through the target expert model.
[0006] In a second aspect, an embodiment of the present application provides a data processing device, comprising: a determination module for determining a target expert model for processing data to be processed; a construction module for constructing the target expert model through shared weights of a first memory and a target low-rank matrix of a second memory, wherein the target low-rank matrix is obtained by pre-decomposing the target expert model; and a processing module for processing the data to be processed through the target expert model.
[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.
[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of the method described in the first aspect.
[0010] In an embodiment of the present application, a target expert model for processing the data to be processed is determined; the target expert model is constructed by the shared weights of the first memory and the target low-rank matrix of the second memory, and the target low-rank matrix is obtained by decomposing the target expert model in advance; the data to be processed is processed by the target expert model, and the target expert model is constructed by the target low-rank matrix when using the corresponding target expert model, which can reduce the use of memory and improve data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application;
[0012] Figure 2 It is a structural schematic diagram of a MOE large model provided in an embodiment of the present application;
[0013] Figure 3 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;
[0014] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0016] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0017] The data processing method, device, electronic device, storage medium and program product provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.
[0018] Figure 1 A data processing method provided by an embodiment of the present application is shown, and the method can be executed by an electronic device. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps:
[0019] Step 102: Determine a target expert model for processing the data to be processed.
[0020] In the embodiment of the present application, a MOE large model is first described in detail. The basic structure of a typical multi-expert MOE large model is as follows: Figure 2 As shown in the figure, the number of experts can be adjusted as a hyperparameter. The MOE large model is generally based on the transformer architecture, by using multiple experts to replace FFN, and then adding a gate as a gating module to control the use of the corresponding experts. Figure 2 As shown, the main function of the layer normalization module (Norm Layer) is to normalize the output. First, the mean and variance of the vector are calculated, and the mean and variance are used to normalize the vector.
[0021] The multi-head attention module is an attention module that connects the outputs of multiple parallel single-head attention modules. Let's first explain the single-head attention module. For the input vector, there is a single matrix Q, K, V. By multiplying the vector with the matrix, we can get the q, k, v vectors. These vectors are used to get the output vector o through the following formula:
[0022]
[0023] Among them, softmax is a common function and d is the length of the vector.
[0024] Feed Forward This module consists of a fully connected network. The input is multiplied by a matrix to get the output.
[0025] For the MOE large model, the main thing is to replace the feedforward network with a MOE feedforward network (MOE FFN). The MOE feedforward network consists of a gating network and an expert network (exp).
[0026] Gated network: It consists of a fully connected network and a softmax structure. The input of the fully connected network is the sequence currently being processed, and the output is a vector of dimension 4. Then, the two positions with the highest probability are selected through softmax, and the corresponding expert network is selected.
[0027] Expert network: The expert network is the same as the ordinary FFN network. This module consists of a fully connected network. The input will be multiplied by a matrix to get the output. The outputs of the two selected expert networks will be added and fused as the final output.
[0028] In the embodiment of the present application, the data set is input into the large model, and the data passes through the attention and gate modules. After passing through the gate, it can be obtained which expert model is assigned to each data to be processed in the data set. The data set passes through the reshape module, and the data to be processed that should be processed by different expert models are concentrated together. For example, the data set includes 6 data to be processed, among which data to be processed 1 is assigned to expert model 1 for processing, data to be processed 2 is assigned to expert model 2 for processing, data to be processed 3 is assigned to expert model 3 for processing, data to be processed 4 is assigned to expert model 4 for processing, data to be processed 5 is assigned to expert model 2 for processing, and data to be processed 6 is assigned to expert model 4 for processing. The reshape module centrally sorts the data to be processed in the order that experts 1-4 should process them respectively, that is, data to be processed 1; data to be processed 2, data to be processed 5; data to be processed 3; data to be processed 4, data to be processed 6. In this way, the currently processed data to be processed and the target expert model used to process the data to be processed can be known.
[0029] Step 104: construct the target expert model through the shared weights of the first memory and the target low-rank matrix of the second memory.
[0030] Before processing the data, you first need to initialize the large model and load the large model into the memory. In the embodiment of the present application, the memory may also refer to the video memory. For multiple expert models that make up the large model, the shared weights of each expert model are the same. At this time, you can apply for 2 blocks of memory with the size of shared weights. The first memory is loaded with shared weights, and the second memory is to be used. In addition, the S / V / D low-rank matrix of each expert model also needs to be loaded into the second memory (video memory), and the memory occupancy of the low-rank matrix can be ignored.
[0031] Since the target expert model is decomposed in advance, the target low-rank matrix is obtained. Therefore, when the data to be processed needs to be processed, the target expert model can be constructed by the shared weights in the first memory and the target low-rank matrix of the target expert model in the second memory.
[0032] Step 106: Process the data to be processed by the target expert model.
[0033] Specifically, after the target expert model is constructed, the data to be processed can be processed by the target expert model.
[0034] In one embodiment, the current expert model i can be determined, where i is [1, 2, 3...n], and n is the number of expert models in the large model. For data that should be processed using expert model i, the low-rank matrix corresponding to expert i in the second memory is used to reassemble the expert model i, and the result of multiplying the low-rank matrix of expert model i is placed in the second memory. The constructed expert model i is used to process the data. After the data processing is completed, the current expert model i=i+1 is determined, and the corresponding data is processed by the next expert model. This step is repeated until all the data to be processed are processed by the corresponding expert model, and the reasoning of the large model is completed.
[0035] After the data to be processed is processed by the target expert model, the method further includes:
[0036] Specifically, after the target expert model processes the data to be processed, the second memory can be released, which reduces memory usage and improves data processing efficiency.
[0037] In one embodiment, for a model with 8x7B parameters, a common MOE large model requires 120GB of video memory. In the embodiment of the present application, without the need for model compression, the video memory occupancy is 15GB, which significantly reduces the video memory occupancy.
[0038] The data processing method provided in the embodiment of the present application determines a target expert model for processing the data to be processed; constructs the target expert model through the shared weights of the first memory and the target low-rank matrix of the second memory, and the target low-rank matrix is obtained by pre-decomposing the target expert model; processes the data to be processed through the target expert model, which can be applied to any MOE large model without the need for model compression or model retraining, and because the second memory only stores low-rank matrices, when using the corresponding target expert model, the target expert model is constructed through the target low-rank matrix, which can reduce memory usage and improve data processing efficiency.
[0039] In one implementation, before determining the target expert model for processing the data to be processed, the method further includes:
[0040] The shared weight is determined according to the weights of multiple expert models, and the multiple expert modules include the target expert model; according to the shared weight, each expert model is decomposed to obtain a low-rank matrix corresponding to each expert model, and the low-rank matrix corresponding to each expert model includes the target low-rank matrix.
[0041] Specifically, before processing the data to be processed, preparation work needs to be done first. The core purpose of the preparation stage is to use multiple low-rank matrices and shared weights to approximate different experts, thereby reducing the memory usage during data processing. For an MOE model containing multiple expert models, the shared weight can be determined based on the weights of the multiple expert models.
[0042] In one implementation, determining the shared weight according to the weights of the multiple expert models includes: determining the average of the weights of the multiple expert models as the shared weight; or determining the weight of any expert model among the multiple expert models as the shared weight.
[0043] Specifically, the average of the weights of multiple expert models can be determined as the shared weight; or the weight of any expert model in the multiple expert models can be determined as the shared weight. Here, taking the average of N expert models as an example, the shared weight is obtained by the following formula:
[0044]
[0045] Among them, E base represents the shared weight, N represents the number of expert models, E i The weight of the expert model.
[0046] After obtaining the shared weights, the shared weights can be used to obtain the additional components of each expert model, and then the additional components are decomposed by SVD. It should be noted that the same operation needs to be performed on multiple experts. The formula for SVD decomposition of the expert model is as follows:
[0047] (E i -E base )=S i V i D i
[0048] After the expert model is decomposed, the low-rank matrix obtained by the decomposition can be stored in the second memory. The memory occupation of the low-rank matrix can be almost negligible, which greatly reduces the memory occupation.
[0049] SVD decomposition to obtain a low-rank matrix includes the following steps:
[0050] Step 1: Calculate the square matrix: First, calculate the product of the transpose of matrix A and A to obtain the n×n square matrix A^TA. Similarly, calculate the product of A and the transpose of A to obtain the m×m square matrix AA^T.
[0051] Step 2, eigenvalue decomposition: Perform eigenvalue decomposition on the matrices A^TA and AA^T. For A^TA, n eigenvalues and corresponding n eigenvectors can be obtained. Similarly, for AA^T, m eigenvalues and corresponding m eigenvectors can also be obtained.
[0052] Step 3: Construct the right singular matrix V: All eigenvectors of A^TA are combined into an n×n matrix V, which is the right singular matrix in the SVD decomposition. Each eigenvector in V is called a right singular vector of A.
[0053] Step 4: Construct the left singular matrix U: Similarly, for the eigenvectors of AA^T, we can form an m×m matrix U, which is the left singular matrix in the SVD decomposition. Each eigenvector in U is called a left singular vector of A.
[0054] Step 5, singular value matrix Σ: The singular value matrix Σ is an m×n diagonal matrix, and the elements on its diagonal are called singular values. These singular values are actually the square roots of the eigenvalues of A^TA and AA^T, and are arranged on the diagonal in descending order. All other off-diagonal elements are 0.
[0055] Step 6: According to the required dimension d, obtain the eigenvectors and eigenvalues corresponding to the largest d singular values, and obtain a low-rank matrix by multiplying them.
[0056] In one implementation, constructing the target expert model by using the shared weights of the first memory and the target low-rank matrix of the second memory includes:
[0057] The target expert model is constructed by using the shared weights, the target singular values in the target low-rank matrix, and the eigenvectors corresponding to the target singular values.
[0058] Specifically, in the preparation stage, singular values and corresponding eigenvectors of rank r can be selected to approximate each expert. r is generally much smaller than the number of rows and columns of the expert matrix, with a typical value of 8 or 16. Each expert matrix will eventually be decomposed into a shared weight matrix plus the product of two low-rank matrices. Since S and D are low-rank matrices, their video memory usage can be almost ignored.
[0059] In this way, when constructing the target expert model, the target expert model can be constructed by sharing weights, target singular values in the target low-rank matrix, and eigenvectors corresponding to the target singular values, as shown in the following formula:
[0060] E i =E base +S ri V ri D ri
[0061] The data processing method provided in the embodiment of the present application does not require model retraining, does not rely on model compression as a pre-step, and can significantly reduce the video memory usage, so it has a wider range of applications and more obvious video memory reduction. The calculation construction is performed only when the expert model is used, which further reduces the video memory usage. Compared with the prior art, the present application does not require data exchange with the disk, and all data exchanges are in the video memory, which has little effect on the inference speed.
[0062] It should be noted that the data processing method provided in the embodiment of the present application can be executed by a data processing device or a control module in the data processing device for executing the data processing method. In the embodiment of the present application, the data processing device provided in the embodiment of the present application is described by taking the execution of the data processing method by the data processing device as an example.
[0063] Figure 3 Schematic diagram of the structure of a data processing device according to an embodiment of the present application. Figure 3 As shown, the data processing device 300 includes: a determination module 310 , a construction module 320 and a processing module 330 .
[0064] A determination module 310 is used to determine a target expert model for processing the data to be processed; a construction module 320 is used to construct the target expert model through the shared weights of the first memory and the target low-rank matrix of the second memory, wherein the target low-rank matrix is obtained by decomposing the target expert model in advance; and a processing module 330 is used to process the data to be processed through the target expert model.
[0065] In one implementation, the construction module 320 is also used to determine the shared weight based on the weights of multiple expert models, and the multiple expert modules include the target expert model; according to the shared weight, each of the expert models is decomposed to obtain a low-rank matrix corresponding to each of the expert models, and the low-rank matrix corresponding to each of the expert models includes the target low-rank matrix.
[0066] In one implementation, the construction module 320 is used to construct the target expert model through the shared weights, the target singular values in the target low-rank matrix, and the eigenvectors corresponding to the target singular values.
[0067] In one implementation, the processing module 330 is further configured to release the second memory.
[0068] In one implementation, the construction module 320 is used to determine the average of the weights of the multiple expert models as the shared weight; or determine the weight of any expert model among the multiple expert models as the shared weight.
[0069] In the embodiments of this application Data processing device It can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., which is not specifically limited in the embodiments of the present application.
[0070] In the embodiments of this application Data processing deviceIt can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.
[0071] The embodiments of the present application provide Data processing device Able to achieve Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0072] Alternatively, if Figure 4 As shown, an embodiment of the present application further provides an electronic device 400, including a processor 401 and a memory 402, wherein the memory 402 stores a program or instruction that can be executed on the processor 401, and when the program or instruction is executed by the processor 401, it implements: determining a target expert model for processing the data to be processed; constructing the target expert model through the shared weights of the first memory and the target low-rank matrix of the second memory, wherein the target low-rank matrix is obtained by pre-decomposing the target expert model; and processing the data to be processed through the target expert model.
[0073] In one implementation, before determining the target expert model for processing the data to be processed, the shared weight is determined according to the weights of multiple expert models, and the multiple expert modules include the target expert model; according to the shared weight, each of the expert models is decomposed to obtain a low-rank matrix corresponding to each of the expert models, and the low-rank matrix corresponding to each of the expert models includes the target low-rank matrix.
[0074] In one implementation, the target expert model is constructed by using the shared weights, the target singular values in the target low-rank matrix, and the eigenvectors corresponding to the target singular values.
[0075] In one implementation, after the data to be processed is processed by the target expert model, the second memory is released.
[0076] In one implementation, the average of the weights of the multiple expert models is determined as the shared weight; or the weight of any expert model among the multiple expert models is determined as the shared weight.
[0077] The specific execution steps can refer to the various steps of the above-mentioned data processing method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.
[0078] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.
[0079] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
[0080] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).
[0081] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.
[0082] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0083] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk.
[0084] An embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the various processes of the above-mentioned data processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0085] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0087] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A data processing method, characterized in that: include: Determining a target expert model for processing the data to be processed; Constructing the target expert model through the shared weights of the first memory and the target low-rank matrix of the second memory, wherein the target low-rank matrix is obtained by decomposing the target expert model in advance; The data to be processed is processed by the target expert model.
2. The method according to claim 1, characterized in that: Before determining the target expert model for processing the data to be processed, the method further includes: Determining the shared weight according to weights of a plurality of expert models, the plurality of expert models including the target expert model; According to the shared weights, each of the expert models is decomposed to obtain a low-rank matrix corresponding to each of the expert models, and the low-rank matrix corresponding to each of the expert models includes the target low-rank matrix.
3. The method according to claim 1, characterized in that The target expert model is constructed by using the shared weights of the first memory and the target low-rank matrix of the second memory, including: The target expert model is constructed by using the shared weights, the target singular values in the target low-rank matrix, and the eigenvectors corresponding to the target singular values.
4. The method according to claim 1, characterized in that: After the data to be processed is processed by the target expert model, the method further includes: Release the second memory.
5. The method according to claim 2, characterized in that: The step of determining the shared weight according to the weights of the plurality of expert models comprises: Determine the average of the weights of the plurality of expert models as the shared weight; or The weight of any expert model among the multiple expert models is determined as the shared weight.
6. A data processing device, characterized in that: include: A determination module, used for determining a target expert model for processing the data to be processed; A construction module, used to construct the target expert model through the shared weights of the first memory and the target low-rank matrix of the second memory, wherein the target low-rank matrix is obtained by decomposing the target expert model in advance; A processing module is used to process the data to be processed through the target expert model.
7. The device according to claim 6, characterized in that The construction module is also used for: Determining the shared weight according to weights of a plurality of expert models, the plurality of expert models including the target expert model; According to the shared weights, each of the expert models is decomposed to obtain a low-rank matrix corresponding to each of the expert models, and the low-rank matrix corresponding to each of the expert models includes the target low-rank matrix.
8. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the data processing method according to any one of claims 1 to 4.
9. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the data processing method according to any one of claims 1 to 4 are implemented.
10. A computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the steps of the data processing method according to any one of claims 1 to 4.
