Data processing method and device and electronic equipment

Abstract

The embodiment of the invention discloses a data processing method and device and electronic equipment. In the method, under the condition that to-be-loaded data is determined, the to-be-loaded data can be partitioned to obtain a plurality of blocks, and the length of each block is determined based on the residual capacity of a first storage area; then, a plurality of blocks can be loaded to the first storage area to be used for training the to-be-trained model, so that the length of each block can be determined according to the residual capacity of the first storage area in the process of partitioning the to-be-loaded data; therefore, the amount of data loaded into the first storage area each time in the data loading process can be determined according to the residual capacity of the first storage area, so that the model training process can better adapt to storage resources in the training equipment, and the storage resources of the training equipment can be used more flexibly.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a data processing method, device, and electronic device. Background Art

[0002] Model training refers to the process of training the parameter weights included in a model (e.g., a large language model) using training data to adjust the model parameters to adapt them to a specific task. For example, in some cases, the model can be fine-tuned so that the fine-tuned large model can adapt to a specific task. However, related model training methods still have the problem of inflexible use of the storage resources of the training equipment. Summary of the Invention

[0003] The present application proposes a data processing method, device and electronic device.

[0004] In a first aspect, the present application provides a data processing method, comprising: dividing the data to be loaded into blocks to obtain multiple blocks, wherein the length of the blocks is determined based on the remaining capacity of a first storage area; loading the multiple blocks into the first storage area for training a model to be trained, wherein the number of blocks loaded into the first storage area each time is a first number.

[0005] In the second aspect, the present application provides a data processing device, which includes: a data processing unit, used to divide the data to be loaded into blocks to obtain multiple blocks, wherein the length of the blocks is determined based on the remaining capacity of the first storage area; a data loading unit, used to load the multiple blocks into the first storage area for training the model to be trained, wherein the number of blocks loaded into the first storage area each time is a first number.

[0006] In a third aspect, the present application provides an electronic device comprising a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above method.

[0007] In a fourth aspect, the present application provides a computer program product, which includes computer instructions stored in a computer-readable storage medium; the computer program product is used to implement the above method.

[0008] In a fifth aspect, the present application provides a computer-readable storage medium, in which program code is stored, wherein the above method is executed when the program code is started and the controller is run.

[0009] The present application provides a data processing method, device and electronic device. In this method, when the data to be loaded is determined, the data to be loaded can be divided into blocks to obtain multiple blocks, wherein the length of the block is determined based on the remaining capacity of the first storage area; then, the multiple blocks can be loaded into the first storage area for training the model to be trained, wherein the number of blocks loaded into the first storage area each time is determined by the first number, so that through the above method, in the process of dividing the data to be loaded into blocks, the length of each block can be determined according to the remaining capacity of the first storage area, and the number of blocks loaded into the first storage area each time is relatively fixed (for example, the first number), thereby realizing that the amount of data loaded into the first storage area each time during the data loading process can be determined according to the remaining capacity of the first storage area, so that the model training process can better adapt to the storage resources in the training device, so as to facilitate more flexible use of the storage resources of the training device. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram showing an application scenario of a data processing method proposed in an embodiment of the present application is shown; Figure 2 A flow chart of a data processing method proposed in an embodiment of the present application is shown; Figure 3 A schematic diagram of dividing data to be loaded into blocks in an embodiment of the present application is shown; Figure 4 A flow chart of a data processing method proposed in another embodiment of the present application is shown; Figure 5 A schematic diagram showing a segmentation method without combining logical breakpoints and a segmentation method with combining logical breakpoints in an embodiment of the present application is shown; Figure 6 A flow chart of a training process involved in an embodiment of the present application is shown; Figure 7 A schematic diagram of decomposing a low-rank matrix in an embodiment of the present application is shown; Figure 8 A structural block diagram of a data processing device proposed in one embodiment of the present application is shown; Figure 9A structural block diagram of an electronic device for executing a data processing method according to an embodiment of the present application is shown; Figure 10 It is a storage unit in an embodiment of the present application for storing or carrying a program code for implementing a data processing method according to an embodiment of the present application; Figure 11 A schematic diagram showing the results of applying multiple training methods on different data sets in an embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0013] Model training is essentially an optimization and adaptation process. It adjusts the model's internal parameters through training data, aiming to enable the model to capture the key features and patterns in the data and adapt to specific task requirements. During the training process, the model will continuously iteratively update the model parameters based on the feedback from the training data to gradually narrow the gap between the predicted results and the actual situation. Among them, fine-tuning as a model training method cleverly utilizes the rich knowledge accumulated by the pre-trained model on general data. It can only fine-tune some of the model's parameters for specific tasks. This not only retains the powerful general capabilities of the pre-trained model, but also can quickly adapt to new tasks, greatly improving the model's development efficiency and application flexibility.

[0014] However, the inventors discovered during their research that the relevant model training methods are not flexible enough in the use of storage resources of the training device (device used to perform model training).

[0015] Therefore, the inventors proposed a data processing method, device and electronic device in the present application that can improve the above-mentioned problems. In this method, when the data to be loaded is determined, the data to be loaded can be divided into blocks to obtain multiple blocks, wherein the length of the block is determined based on the remaining capacity of the first storage area; then, the multiple blocks can be loaded into the first storage area for training the model to be trained, wherein the number of blocks loaded into the first storage area each time is determined by the first number, so that through the above method, in the process of dividing the data to be loaded into blocks, the length of each block can be determined according to the remaining capacity of the first storage area, and the number of blocks loaded into the first storage area each time is relatively fixed (for example, the first number), thereby realizing that the amount of data loaded into the first storage area each time during the data loading process can be determined according to the remaining capacity of the first storage area, so that the model training process can better adapt to the storage resources in the training device, so as to facilitate more flexible use of the storage resources of the training device.

[0016] The following first introduces the application scenarios involved in the embodiments of this application.

[0017] In an embodiment of the present application, the data processing method provided can be executed by a server. In this way executed by the server, the server can start executing the steps in the data processing method provided in the embodiment of the present application in response to a trigger instruction. The trigger instruction can be sent by an electronic device used by the user, or it can be triggered locally by the server in response to some automated events. For example, in the case where the electronic device sends a user verification request to the server, the server can start executing the data processing method in response to receiving the user verification request, wherein the user verification request can carry information of the user to be identified.

[0018] In addition, if Figure 1 As shown, the data processing method provided by the embodiment of the present application can also be performed collaboratively by an electronic device and a server. In this manner performed collaboratively by an electronic device and a server, some of the steps in the data processing method provided by the embodiment of the present application are performed by the electronic device, while the other steps are performed by the server. Exemplarily, the electronic device 100 can perform the data processing method including: dividing the data to be loaded into blocks to obtain a plurality of blocks. Afterwards, the electronic device 100 transmits the plurality of blocks to the server 200, and the server 200 then performs subsequent steps to obtain a trained model. In the case where the server 200 obtains the trained model, the trained model can be retained locally, or sent to other servers for operation, or it can be sent to the electronic device 100 for operation in the electronic device 100.

[0019] It should be noted that in this method of collaborative execution by the electronic device and the server, the steps respectively executed by the electronic device and the server are not limited to the methods introduced in the above examples. In actual applications, the steps respectively executed by the electronic device and the server can be dynamically adjusted according to actual conditions.

[0020] It should be noted that the electronic device 100 is Figure 1 and Figure 2 In addition to the smartphone shown in the figure, it can also be a tablet computer, smart glasses and other devices. Server 200 can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers. In the case where the data processing method provided in the embodiment of the present application is executed by a server cluster or distributed system composed of multiple physical servers, different steps in the data processing method can be executed by different physical servers respectively, or can be executed in a distributed manner by servers built based on the distributed system.

[0021] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0022] See also Figure 2 , an embodiment of the present application provides a data processing method, the method comprising: S110: Divide the data to be loaded into blocks to obtain a plurality of blocks, wherein the length of the blocks is determined based on the remaining capacity of the first storage area.

[0023] In the embodiment of the present application, the data to be loaded can be understood as data for recording in the first storage area for use in training the model to be trained. The first storage area can be the memory of the training device (a device for executing the data processing method), or it can be the memory of the processor in the training device for executing the data processing method. For example, in the case where the data processing method is executed by the graphics card in the training device, the first storage area can be the memory (video memory) of the graphics card.

[0024] In the embodiments of the present application, the model to be trained can be understood as a model that has not been pre-trained, or the model to be trained can be understood as a pre-trained model. In which case, when the model to be trained is a pre-trained model, the data to be loaded involved in the embodiments of the present application can be understood as data for fine-tuning the model to be trained.

[0025] Optionally, the model to be trained involved in the embodiment of the present application can be a pre-trained large language model (Large Language Model, LLM), also referred to as a large model. The large language model is a natural language processing technology based on deep learning. It is trained with massive text data and has the ability to generate, understand and reason natural language. Its core adopts the Transformer architecture, which uses the self-attention mechanism to capture long-distance semantic associations, and realizes the leap from general language law learning to vertical field task adaptation. Among them, in the case where the model to be trained is a large language model, the process of training the model to be trained can be understood as a process of fine-tuning a large model. Among them, large model fine-tuning is a migration learning technology based on a pre-trained model, and its core idea is to use the knowledge learned by the pre-trained model on massive general data, and further train it on a small-scale data set for a specific task to adjust some parameters of the model (such as the weight of the last layer or a specific layer) to adapt it to the needs of the new task. This approach avoids the high cost of training the model from scratch while retaining the powerful general capabilities of the pre-trained model.

[0026] Among them, for the neural network model (for example, a large model), an input limit can be configured. If the input data is too long, direct input will result in truncation or an error. Therefore, when the data to be loaded is obtained, the data to be loaded can be first divided into blocks, so as to gradually load the data to be loaded into the first storage area in the form of blocks. Among them, dividing the data to be loaded into blocks can be understood as dividing the data to be loaded into blocks to obtain multiple blocks. The multiple blocks can be 2, 3 or even more.

[0027] In the process of training the model to be trained, the data input into the model to be trained will be loaded into the first storage area. Therefore, the process of loading the data to be loaded into the first storage area can be understood as the process of inputting the data to be loaded into the model to be trained. In addition to storing the data used for training, the first storage area also stores other data. For example, it can also be used to store parameters included in the model to be trained itself, or it can be used by the device (device executing the data processing method) to store data for other tasks being run. Therefore, by determining the length of the block based on the remaining capacity of the first storage area, the amount of data loaded into the first storage area each time can be adapted to the current remaining capacity of the first storage area or to the current task running status of the device. In the process of training the model to be trained, the data input into the first storage area can be loaded into the first storage area each time. For example, if one block is loaded into the first storage area each time, the amount of data loaded can be understood as the size of one block. For another example, if two blocks are loaded each time, the amount of data loaded can be understood as the size of two blocks.

[0028] Among them, adapting to the current remaining capacity of the first storage area can be understood as loading relatively less data into the first storage area each time data is loaded when the remaining capacity of the first storage area is small, so as to avoid occupying the remaining space of the first storage area too quickly. Correspondingly, when the remaining capacity of the first storage area is large, relatively more data can be loaded into the first storage area each time data is loaded, so as to improve the utilization rate of the first storage area. Among them, in order to achieve the aforementioned adaptation to the current remaining capacity of the first storage area, as a way, when the remaining capacity of the first storage area is greater than a preset capacity threshold, the length of the block can be configured to a first length; when the remaining capacity of the first storage area is less than the preset capacity threshold, the length of the block can be configured to a second length, and the first length is greater than the second length.

[0029] The preset capacity threshold can be configured in advance by the developer, or can be dynamically configured according to actual conditions.

[0030] It should be noted that when the training device (device for executing the data processing method) is in a busy state, it indicates that the training device currently has more tasks to perform (for example, in addition to being used to execute the training tasks corresponding to the data processing method, it is also used to execute other tasks), or the amount of calculation to be executed is large. Therefore, the probability that the first storage area is used or the probability that the first storage area will be used is large. Correspondingly, when the training device is in a non-busy state, it indicates that the training device currently has fewer tasks to perform (for example, only used to execute the training tasks corresponding to the data processing method), or the amount of calculation to be executed is small. Therefore, the probability that the first storage area is used or the probability that the first storage area will be used is small.

[0031] As a method, a relatively large preset capacity threshold can be configured when the training device is in a busy state, so that it is not easy to trigger the increase of the length of the block. A relatively small preset capacity threshold can be configured when the training device is in a non-busy state, so that it is relatively easier to trigger the increase of the length of the block. For example, when the training device is in a busy state, the preset capacity threshold configured is a first threshold, and when the training device is in a non-busy state, the preset capacity threshold configured is a second threshold, wherein the first threshold is greater than the second threshold.

[0032] As a method, whether the training device is busy or not can be determined based on the current processor occupancy of the training device. It should be noted that during the model training process, the processor of the training device undertakes a large number of computing tasks. By monitoring the current processor occupancy of the training device, you can intuitively understand the processor workload. When the processor occupancy rate is high, it means that the processor is processing a large number of tasks, that is, the device is busy. At this time, there may not be enough computing resources to efficiently process new training tasks or perform other operations; when the processor occupancy rate is low, it means that the processor has more idle resources, that is, the device is not busy and can handle new tasks more easily. For example, if the processor occupancy rate is continuously above 90%, it usually indicates that the training device is busy; if the GPU occupancy rate is less than 30%, it indicates that the training device is not busy.

[0033] For example, taking the first storage area as video memory (graphics card memory), when there is a lot of remaining video memory, the block length is extended from 512 to 1024 tokens to maximize the use of hardware resources. The formula is: in, The maximum input length of the model.

[0034] S120: Load the multiple blocks into the first storage area for training the to-be-trained model, wherein the number of blocks loaded into the first storage area each time is a first number.

[0035] In the embodiment of the present application, there are multiple ways to load the blocks obtained by segmentation. As one way, the blocks obtained by segmentation can be loaded according to the first quantity after the segmentation begins, without waiting for the data to be loaded to be completely segmented before loading the data. As another way, the multiple blocks obtained by segmentation can be loaded after the data to be loaded is completely segmented. For example, taking the example of loading one block to the first storage area each time, as shown in FIG. Figure 3 As shown in FIG, if the data to be loaded is completely divided into blocks, block B1, block B2, and block B3 can be obtained. As a method, when block B1 is obtained (block B1 is the first block obtained), block B1 can be loaded into the first storage area without waiting for blocks B2 and B3 to be obtained before loading data.

[0036] This embodiment provides a data processing method, in which, when the data to be loaded is determined, the data to be loaded can be divided into blocks to obtain multiple blocks, wherein the length of the blocks is determined based on the remaining capacity of the first storage area; then, the multiple blocks can be loaded into the first storage area for training the model to be trained, wherein the number of blocks loaded into the first storage area each time is determined by the first number, so that through the above method, in the process of dividing the data to be loaded into blocks, the length of each block can be determined according to the remaining capacity of the first storage area, and the number of blocks loaded into the first storage area each time is relatively fixed (for example, the first number), thereby realizing that the amount of data loaded into the first storage area each time during the data loading process can be determined according to the remaining capacity of the first storage area, so that the model training process can better adapt to the storage resources in the training device, so as to facilitate more flexible use of the storage resources of the training device.

[0037] See also Figure 4 , an embodiment of the present application provides a data processing method, the method comprising: S210: Divide a data set for training into multiple parts, wherein the data set includes multiple documents and each part includes at least one document.

[0038] The training dataset may include multiple documents. The documents in the dataset can be divided into multiple sections based on their content, so that the documents in each section have a certain degree of content relevance. Optionally, the documents in each section may belong to the same field or have similar content. For example, all medical documents may be placed in one section, and all financial reports in another.

[0039] As a way to calculate the content similarity between two documents, it can be achieved by extracting document features (such as keywords, phrases, or topic distribution) and then applying a text similarity algorithm. Available methods include Jaccard similarity based on vocabulary overlap, cosine similarity that measures the angle between vectors, edit distance that calculates the number of edit operations, and pre-trained language models (such as BERT) that use deep learning models to capture semantic information.

[0040] S220: Loading metadata of each of the plurality of documents into a first storage area.

[0041] In the embodiments of the present application, the metadata of a document does not directly constitute the main content of the document, but rather provides additional information about the document to help users or systems better understand, manage, and use the document. Optionally, the metadata of a document is data that describes the document's attributes (e.g., the document's title, author, creation date, modification date, file size, file type (e.g., PDF, Word document, etc.), structure (e.g., chapter titles, hierarchical relationships, page numbers, etc.), source, and management information.

[0042] S230: Determine data to be loaded from the multiple parts according to current training requirements.

[0043] Among them, the training demand can also be understood as the current training goal. Optionally, the training demand can be determined based on the task to be performed by the model obtained by training the training model. This means that the current training demand characterizes the task to be performed by the trained model. For example, if the task to be performed by the trained model is a formula recognition task, then the data to be loaded determined from multiple parts can be data related to the formula. For another example, if the task to be performed by the trained model is a content summarization task, then the data to be loaded determined from multiple parts can be data related to summary content (for example, abstract).

[0044] Optionally, the training requirements represent the functionality of the model that needs to be trained.

[0045] S240: Divide the data to be loaded into blocks to obtain a plurality of blocks, wherein the length of the blocks is determined based on the remaining capacity of the first storage area.

[0046] S250: Load the multiple blocks into the first storage area for training the to-be-trained model, wherein the number of blocks loaded into the first storage area each time is a first number.

[0047] Optionally, the first quantity can be pre-configured by the developer, or can be dynamically updated based on actual conditions. For example, the first quantity can be dynamically configured based on the remaining capacity of the first storage area. If the remaining capacity of the first storage area is large, a larger first quantity can be configured; if the remaining capacity of the first storage area is small, a smaller first quantity can be configured. Exemplarily, the first quantity is 1 or 2, etc.

[0048] The present embodiment provides a data processing method, so that in the process of dividing the data to be loaded into blocks, the length of each block can be determined according to the remaining capacity of the first storage area, and the number of blocks loaded into the first storage area each time is relatively fixed (for example, the first number), thereby realizing that the amount of data loaded into the first storage area each time during the data loading process can be determined according to the remaining capacity of the first storage area, so that the model training process can better adapt to the storage resources in the training device, so as to facilitate more flexible use of the storage resources of the training device. Moreover, in the present embodiment, the metadata of each document in the data set used for training can be first loaded into the first storage area for training the model to be trained, and then the data to be loaded can be determined from multiple parts according to the current training needs during the training process, so that the data record has better pertinence, and avoids loading too much data that is irrelevant to the current training needs and affecting the model training effect.

[0049] It should be noted that, in the embodiment of the present application, the data to be loaded can be understood as a document, a video or an image. In this case, the data to be loaded can be quantified by quantity. Correspondingly, in the embodiment of the present application, multiple data to be loaded can be synchronously divided into blocks. Correspondingly, in the process of loading the blocks, it can be understood that multiple blocks corresponding to the multiple data to be loaded are loaded into the first storage area. In this way, the number of blocks loaded into the first storage area each time is the second number. Among them, the second number corresponds to the number of data to be loaded that are processed in parallel. If two data to be loaded (for example, two documents) are currently being processed, then the second number can be 2. If three data to be loaded (for example, two documents) are currently being processed, then the second number can be 3.

[0050] As a method, in an embodiment of the present application, during the process of segmenting the data to be loaded, logical breakpoints in the data to be loaded can be further identified. Taking the data to be loaded as a document as an example, the logical breakpoints can include paragraph boundaries, punctuation marks, and entity boundaries. The entity can be an image in the document, for example. In this manner, when segmenting the data to be loaded, the length of the resulting segment is determined by combining the remaining capacity of the first storage area and the logical breakpoints.

[0051] Optionally, during the block segmentation process, the data to be loaded can be segmented from the starting position of the data to be loaded toward the ending position of the data to be loaded, wherein content recognition can be performed from the previous segmentation point (or the starting position of the data to be loaded) toward the ending position of the data to be loaded. If a logical breakpoint is preferentially identified, a segmentation is performed at the identified logical breakpoint, and the subsequent segmentation process is continued. If no logical breakpoint is identified between the previous segmentation point and the target position, a segmentation is performed at the target position, wherein the length between the previous segmentation point and the target position is the preset block length.

[0052] For example, Figure 5 As shown, in Figure 5 The difference between the two segmentation methods without and with logical breakpoints is shown in Figure 2. Figure 5 The method shown in FIG. 1 is not combined with the logical breakpoint. In this method, the determined segmentation positions W11 and W12 are both determined based on the preset length of the segmentation blocks.

[0053] For example, splitting at the splitting position W11 may obtain block B1, splitting at the splitting position W12 may obtain block B2, and the remaining portion of the data to be loaded may obtain block B3.

[0054] In this segmentation method combined with logical breakpoints, in addition to identifying the content length, logical breakpoints will also be identified. For example, if position W21 is identified as a logical breakpoint, position W21 will be determined as a segmentation position for segmentation. Among them, if there is no logical breakpoint between position W21 and position W11, and the content length between position W21 and position W11 does not reach the length of the block, segmentation will not be performed at position W11. Afterwards, if it is identified that the length of the content between position W22 and position W21 meets the length of the block, position W22 will be determined as the target position, that is, a segmentation will be performed at position W22. Similarly, a segmentation will be performed at position W23 later. Among them, segmentation at position W21 can obtain block B4, segmentation at position W22 can obtain block B5, segmentation at position W23 can obtain block B6, and for the remaining part of the data to be loaded, block B7 can be obtained.

[0055] Among them, by combining logical breakpoints to divide the blocks, it is possible not only to make the loading of the data to be loaded adaptable to the current remaining capacity of the first storage area, but also to make the obtained blocks a semantically complete content, which is conducive to the model to be trained to process the training data more accurately, so as to improve the model training effect.

[0056] As a method, the data processing method provided in the embodiment of the present application may also include: in the process of training the model to be trained, storing the first data in the activation value of the model to be trained in the first storage area, and storing the second data in the activation value in the second storage area, wherein the access frequency of the first data is higher than the access frequency of the second data, and the data reading speed of the first storage area is faster than the data reading speed of the second storage area.

[0057] During the forward propagation process of model training, activation values ​​are involved. The activation value is the output value calculated by the activation function (such as ReLU, Sigmoid, Tanh, etc.) after the neurons in the neural network perform weighted summation on the input signals. It represents the activity level of the neuron under the current input. For example, when the model to be trained is a pre-trained model, the activation value of the pre-trained model already contains rich common features (such as word meaning and syntactic structure in the language model). Through fine-tuning, the activation value can be made more suitable for the target task, that is, the fine-tuned activation value will gradually tend to the characteristics of the target task. For example, when the target task is a sentiment analysis task, the activation values ​​of some neurons may be more sensitive to sentiment words (such as "good" and "bad").

[0058] Activation values ​​can also be stored in the first storage area. However, as mentioned above, the overall space of the first storage area is limited. If all activation values ​​are directly stored in the first storage area, it may cause significant storage pressure. Therefore, by identifying the first data and the second data and storing only the first data in the first storage area, it is beneficial to reduce the occupancy of the first storage area without significantly affecting access to activation values.

[0059] As a method, the processing method of the activation value in the embodiments of the present application can be understood as a dynamic cache activation method. Dynamic cache activation is a method of dynamically adjusting the cache strategy based on real-time data access patterns and system status. The content involved includes: (1) Data access pattern prediction, wherein the data access frequency, sequentiality (e.g., sequential / random access), and temporal locality characteristics of the data in the activation value can be monitored in real time through machine learning models or time series analysis to predict future hot data in the activation value. For example, an RNN (Recurrent Nearal Networks) model can process the access log sequence of an application and output the hit / miss probability of the next access. Furthermore, data with a high access frequency is determined as the aforementioned first data, and data with a low access frequency is determined as the aforementioned second data.

[0060] (2) Dynamic resource allocation, where high-frequency access data (e.g., the first data mentioned above) is retained in a fast storage layer (such as memory or SSD (Solid State Drive Cache) cache), and low-frequency data (e.g., the second data mentioned above) is downgraded to a slow storage layer. The fast storage layer can be understood as the first storage area mentioned above, and the satisfied storage layer can be understood as the second storage area mentioned above. The second storage area can be main memory (RAM), where main memory can also be understood as the memory of the training device.

[0061] (3) Divide the storage area into multiple sub-areas (such as hotspot area and cold data area), and adjust the capacity and replacement strategy of each partition as needed, such as LRU (Least Recently Used) or LFU (Least Frequently Used) strategies. As a way, the first storage area can be divided into a hotspot area and a cold data area, wherein the hotspot area is used to store data with a relatively high access frequency, and the cold data area can be used to store data with a relatively low access frequency. In addition, the data in the hotspot area, the cold data area and the second storage area can be managed according to the access frequency of the data. For example, when the access frequency of some data in the cold data area increases, the part of the data can be further moved to the hotspot area. Correspondingly, when the access frequency of some data in the cold data area decreases, the part of the data can be further moved to the second storage area.

[0062] (4) Real-time invalidation and update mechanism. When the data source changes (such as a database update) or the system load changes, an event-driven mechanism is used to trigger cache invalidation or proactively preload new data, solving the lag problem of traditional caches. For example, by combining Memcached callback functions, the cache can be updated synchronously when data is written. In practical applications, this can be combined with Gradient Checkpointing.

[0063] As one approach, in the data processing method provided in an embodiment of the present application, loading the multiple blocks into the first storage area for training the model to be trained may include: The multiple blocks are loaded into the first storage area for parameter adjustment of the low-rank matrix (Low-rank Matrix) corresponding to the model to be trained, so as to train the model to be trained through the low-rank matrix after the parameter adjustment; wherein the rank of the low-rank matrix is ​​determined by the importance score of the low-rank matrix. It should be noted that the rank of a matrix is ​​the maximum number of linearly independent row vectors or column vectors in the matrix, which reflects the "effective dimension" of the information contained in the matrix. A low-rank matrix refers to a matrix whose rank is much smaller than its number of rows and columns. For example, an m×n (m is the number of rows, n is the number of columns) matrix, if its rank r satisfies , then this matrix is ​​a low-rank matrix.

[0064] It should be noted that in one approach, in order to avoid performing a full parameter update on the model to be trained, low-rank decomposition (Low-Rank Decomposition) can be used to perform parameter-efficient fine-tuning (PEFT) on the model to be trained. The main principle of using low-rank decomposition to perform efficient parameter fine-tuning on the model to be trained is that the parameter update of the model (e.g., the model to be trained) on a specific task can be approximated as a linear combination of low-rank matrices, and the model can then be fine-tuned using the low-rank matrix. The specific implementation is as follows: (1) Freeze pre-training parameters: keep the original model weight matrix unchanged to avoid catastrophic forgetting; (2) Introducing low-rank adapters: Add two trainable low-rank matrices next to the weights of the Transformer layer and , where rank .

[0065] (3) Incremental update simulation: through , approximate the parameter update amount of full fine-tuning, and the final output is .

[0066] like Figure 6 As shown, the mathematical form is: This design makes the number of training parameters from down to For example, in the 175B parameter GPT-3, LoRA only needs to adjust 0.01% of the parameters (about 175 million) to complete task adaptation.

[0067] It should be noted that in multi-task or continuous learning scenarios, the rank requirements of different tasks may change dynamically. When the task complexity is high (such as cross-modal learning and mathematical reasoning), fixed rank may not be able to capture sufficient feature interactions, resulting in performance significantly lower than full parameter fine-tuning. For example, code generation tasks require high-rank matrices to simulate complex logical structures, and a fixed rank of r = 8 may not cover enough parameter space.

[0068] Therefore, the embodiment of the present application provides a dynamic rank adjustment mechanism. The core of the dynamic rank adjustment mechanism is to dynamically adjust the rank of the low-rank matrix according to the importance of the weight matrix, breaking the limitation of the fixed rank of the low-rank matrix in the related method.

[0069] Optionally, as a way to determine the importance score, the low-rank matrix can be decomposed into multiple matrices, the multiple matrices including a first orthogonal matrix, a second orthogonal matrix and a diagonal singular value matrix; the importance score of the low-rank matrix can be obtained by the importance score of each feature in the multiple matrices, wherein the importance score of each feature is obtained by the feature sensitivity and feature uncertainty of each feature.

[0070] Among them, the low rank matrix can be decomposed by singular value decomposition (SVD). As a way, singular value decomposition is as follows Figure 7 As shown. Among them, Figure 7 As shown in the figure, the wireframe represents the feature obtained by decomposing a feature in the low-rank matrix. The leftmost matrix is ​​the first orthogonal matrix, the middle matrix is ​​the diagonal singular value matrix, and the rightmost matrix is ​​the second orthogonal matrix.

[0071] For the decomposed matrix, P represents the first orthogonal matrix, Characterize the diagonal singular value matrix, Representing the second orthogonal matrix. The importance score of a single feature can be expressed as: in, represents the importance score of the i-th triplet of the k-th module, Represents the kth module The i-th element of the matrix, represents the j-th row and i-th column element of the P matrix of the k-th module, Represents the i-th row and j-th column element of the Q matrix of the k-th module.

[0072] In the above formula for importance score, S(*) represents the importance of a single feature (or can be understood as an importance score). The importance of a single feature can take into account both feature sensitivity and feature uncertainty. The detailed calculation formula is as follows: Sensitivity: For a single parameter in the A / B matrix , its sensitivity is defined as the product of the feature weight and the absolute value of the gradient (or, feature sensitivity refers to the degree of influence of a single parameter (or feature) in the model on the model output), and the expression formula is: in, are any trainable weights, is the weight gradient.

[0073] Because in stochastic gradient descent (SGD), this importance is only the importance of the sample response of a single batch, so optionally, a sliding average (EMA) can be used to reduce the importance error of a single batch. The expression formula is as follows: It is a constant hyperparameter, generally set between 0.7 and 0.9. The uncertainty of a feature represents the local temporal variation in sensitivity. During model training, feature sensitivity may change as training progresses. This change reflects the model's continuous adjustment of its understanding of feature importance. Feature uncertainty is quantified by calculating the change in sensitivity over time, reflecting the model's uncertainty about feature importance. The formula for expressing feature uncertainty is: Also apply the sliding average: It is a constant hyperparameter, generally set between 0.8 and 0.95.

[0074] As a way, after calculating the importance of the triples, the total parameter budget b(t) can be dynamically adjusted during the training phase to resize the rank of the low-rank matrix. The details of the total parameter budget b(t) over the training time can be as follows: Initial stage ( ): b(t) grows linearly, exploring high-rank possibilities; mid-term ( ): Stabilize the budget and optimize allocation; Late ): Fixed budget, only keeping high-importance parameters.

[0075] That is to say, in the early stage of training, you can try to configure a relatively higher rank for the low-rank matrix, and as the training progresses, you can reduce the rank of the low-rank matrix, so that the rank of the low-rank matrix can change from high to low during the training stage.

[0076] The calculation formula for b(t) is as follows: Optionally, you can periodically trim low-scoring parameters, keeping only the first b(t) important parameters and setting all others to 0.

[0077] Optionally, the rank of the low-rank matrix is ​​determined by the importance score of the low-rank matrix and the task for which the model is to be trained. In this way, the rank of the low-rank matrix can be determined jointly by the importance score and the task for which the model is to be trained.

[0078] Among them, as a way, when the rank of the low-rank matrix is ​​determined by the importance score, a first correspondence between the importance score and the rank can be established, so that after the importance score of the low-rank matrix is ​​determined, the rank of the low-rank matrix can be determined according to the first correspondence. Optionally, the specific method of establishing the first correspondence may vary depending on the application scenario and specific needs. As a way, a large number of low-rank matrix samples can be collected first, and the importance score and corresponding rank can be determined for each low-rank matrix sample. Then, through statistical analysis, machine learning and other methods, the potential rules between the importance score and the rank are excavated, so as to establish a mathematical model or mapping relationship between the two, that is, the first correspondence. For example, a regression analysis method can be used to fit the functional relationship between the importance score and the rank, or a classification algorithm can be used to divide the importance score into different intervals, each interval corresponding to a specific rank range.

[0079] Among them, for low-rank matrices with higher importance scores, the corresponding rank will be relatively higher. Among them, low-rank matrices with high importance scores usually contain more key information and require more linearly independent rows or columns to accurately represent this information, so their rank is relatively high.

[0080] Then, when the rank of the low-rank matrix is ​​determined by the importance score and the task for which the model is to be trained, a second correspondence between the total score and the rank can be established. The total score is obtained by the importance score of the low-rank matrix and the score corresponding to the task for which the model is to be trained. For example, the importance score of the low-rank matrix and the score corresponding to the task for which the model is to be trained can be added to obtain the total score. Correspondingly, for a low-rank matrix with a higher total score, the corresponding rank is higher.

[0081] As a method for determining the second correspondence, it includes data collection and preparation as well as model training and fitting. Data collection and preparation involves collecting a large number of low-rank matrix samples from different scenarios, determining for each low-rank matrix sample its importance score, the corresponding task, and a reasonable rank determined through other methods (such as expert experience or actual performance evaluation). Furthermore, a corresponding score is set for each task. Model training and fitting involves using machine learning algorithms, such as regression analysis and neural networks, to train the model using the total score as input and the rank of the low-rank matrix as output. By continuously adjusting the model parameters, the model accurately fits the relationship between the total score and rank, thereby establishing a mathematical model between the total score and rank, i.e., the second correspondence. In practical applications, when faced with a new low-rank matrix and application task, the importance score of the low-rank matrix and the score corresponding to the application task are first calculated, and then the total score is obtained according to the above method. Finally, the total score is substituted into the second correspondence to determine the rank of the low-rank matrix.

[0082] It should be noted that when the model to be trained includes multiple layers, there can be a corresponding low-rank matrix for each layer. Therefore, the corresponding rank can be dynamically determined for the low-rank matrix of each layer.

[0083] In an embodiment of the present application, the learning rate determines the step size of the weights and biases adjusted according to the gradient of the loss function during the training process of the model. Specifically, the learning rate can be used to control the amplitude of each parameter update. A suitable learning rate can enable the model to converge quickly to the optimal solution during the training process. If the learning rate is too large, the model may oscillate near the optimal solution or even fail to converge; if the learning rate is too small, the model convergence speed will be too slow, increasing the training time. As a way, in the data processing method provided in an embodiment of the present application, the learning rate can be adjusted by cosine curvature during the training of the training model.

[0084] Optionally, in each training cycle, the learning rate is adjusted according to the cosine curvature as follows: in, 、 are the upper and lower bounds of the learning rate, is the current cycle length, is the current step number.

[0085] In an embodiment of the present application, the learning rate can slowly decrease from the initial value, accelerate in the middle stage, and slow down again in the end stage, which helps to fine-tune the parameters. Among them, adjusting the learning rate by cosine curvature, that is, the cosine annealing learning rate adjustment method, is based on the mathematical properties of the cosine function. Among them, the cosine function has the characteristics of smoothness and periodicity, and the learning rate will change along the cosine curve. During the training process, the learning rate starts from the maximum value and gradually decreases to the minimum value as the training cycle progresses. Specifically, as the independent variable in the cosine function increases, the function value first slowly decreases, then accelerates and finally slowly decreases again. This decline pattern is consistent with the adjustment requirements of the learning rate. In the early stages of training, using a larger learning rate can make the model parameters quickly adjusted, approach the global minimum more quickly, and accelerate the convergence process of the model. In the later stages of training, a smaller learning rate can fine-tune the model, improve the stability and generalization ability of the model, and make the model perform better on the test set.

[0086] The training process may also include warm restarts. step), the learning rate is reset to the initial value and a new cosine annealing cycle begins. When restarting, the cycle length gradually increases: for example = , making later training more stable. Parameter inheritance is also possible during a hot restart, that is, the model parameters are inherited from the end of the previous cycle, avoiding performance fluctuations caused by cold starts.

[0087] See also Figure 8 , an embodiment of the present application provides a data processing device 300, the device 300 comprising: The data processing unit 310 is configured to divide the data to be loaded into blocks to obtain a plurality of blocks, wherein the length of the blocks is determined based on the remaining capacity of the first storage area.

[0088] The data loading unit 320 is used to load the multiple blocks into the first storage area for training the model to be trained, wherein the number of blocks loaded into the first storage area each time is a first number.

[0089] As a method, the data processing unit 310 is also used to configure the length of the block to a first length when the remaining capacity of the first storage area is greater than a preset capacity threshold; and to configure the length of the block to a second length when the remaining capacity of the first storage area is less than the preset capacity threshold, and the first length is greater than the second length.

[0090] As a method, the data processing unit 310 is also used to divide the data set for training into multiple parts, wherein the data set includes multiple documents and each part includes one document; load the metadata of each of the multiple documents into a first storage area; and determine the data to be loaded from the multiple parts according to current training requirements.

[0091] As a method, the data loading unit 320 is also used to store the first data in the activation value of the model to be trained in the first storage area, and store the second data in the activation value in the second storage area during the training of the model to be trained, wherein the access frequency of the first data is higher than the access frequency of the second data, and the data reading speed of the first storage area is faster than the data reading speed of the second storage area.

[0092] As a method, the data loading unit 320 is specifically used to load the multiple blocks into the first storage area for performing parameter adjustment on the low-rank matrix corresponding to the model to be trained, so as to train the model to be trained by performing the parameter-adjusted low-rank matrix; wherein the rank of the low-rank matrix is ​​determined by the importance score of the low-rank matrix.

[0093] Optionally, the data loading unit 320 is specifically used to decompose the low-rank matrix into multiple matrices, wherein the multiple matrices include a first orthogonal matrix, a second orthogonal matrix and a diagonal singular value matrix; the importance score of the low-rank matrix is ​​obtained by the importance score of each feature in the multiple matrices, wherein the importance score of each feature is obtained by the feature sensitivity and feature uncertainty of each feature.

[0094] This embodiment provides a data processing device, so that through the above-mentioned data processing device, in the process of dividing the data to be loaded into blocks, the length of each block can be determined according to the remaining capacity of the first storage area, and the number of blocks loaded into the first storage area each time is relatively fixed (for example, the first number), thereby realizing that the amount of data loaded into the first storage area each time during the data loading process can be determined according to the remaining capacity of the first storage area, so that the model training process can better adapt to the storage resources in the training device, so as to facilitate more flexible use of the storage resources of the training device.

[0095] It should be noted that the device embodiment in this application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0096] The following will be combined Figure 9 An electronic device provided by this application is described.

[0097] See also Figure 9 Based on the above flowchart processing method and device, the embodiment of the present application also provides an electronic device 1000 that can execute the above flowchart processing method. The electronic device 1000 includes one or more (only one is shown in the figure) processors 105, a memory 104, an audio playback module 106, and an audio acquisition device 108 that are coupled to each other. The memory 104 stores a program that can execute the content of the above embodiment, and the processor 105 can execute the program stored in the memory 104.

[0098] The processor 105 may include one or more processing cores. The processor 105 utilizes various interfaces and circuits to connect various components within the electronic device 1000. It executes instructions, programs, code sets, or instruction sets stored in the memory 104, and accesses data stored in the memory 104 to perform various functions and process data within the electronic device 1000. Optionally, the processor 105 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 105 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 105 and may be implemented separately via a communication chip.

[0099] The memory 104 may include a random access memory (RAM) or a read-only memory (ROM). The memory 104 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.

[0100] Furthermore, in addition to the aforementioned components, the electronic device 1000 may further include a network module 110 and a sensor module 112 .

[0101] The network module 110 is used to realize information interaction between the electronic device 1000 and other devices. For example, a connection can be established with other audio playback devices or other electronic devices, and information interaction is performed based on the established connection. As a way, the network module 110 of the electronic device 1000 is a radio frequency module, which is used to receive and send electromagnetic waves, realize the mutual conversion of electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The radio frequency module may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM) cards, memories, etc. For example, the radio frequency module can interact with external devices by sending or receiving electromagnetic waves.

[0102] The sensor module 112 may include at least one sensor. Specifically, the sensor module 112 may include, but is not limited to, a pressure sensor, a motion sensor, an acceleration sensor, and other sensors.

[0103] Among them, the pressure sensor can be a sensor that detects pressure generated by pressing on the electronic device 1000. That is, the pressure sensor detects pressure generated by contact or pressing between the user and the electronic device 1000, for example, pressure generated by contact or pressing between the user's ear and the electronic device 1000. Therefore, the pressure sensor can be used to determine whether contact or pressing occurs between the user and the electronic device 1000, and the magnitude of the pressure.

[0104] The accelerometer can detect the magnitude of acceleration in all directions (generally three axes) and the magnitude and direction of gravity when stationary. This can be used for applications that recognize the posture of the electronic device 1000 (such as switching between landscape and portrait modes, related games, and magnetometer posture calibration), vibration recognition-related functions (such as pedometers and tapping), etc. In addition, the electronic device 1000 may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, and thermometer, which will not be detailed here.

[0105] The audio collection device 108 is used to collect audio signals. Optionally, the audio collection device 108 includes multiple audio collection devices, which may be microphones.

[0106] Please refer to Figure 10 , which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 1100 stores program code, which can be called by a processor to execute the method described in the above method embodiment.

[0107] Computer-readable storage medium 1100 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Alternatively, computer-readable storage medium 1100 may include a non-transitory computer-readable storage medium. Computer-readable storage medium 1100 may have a storage area for program code 1110 for executing any of the method steps described above. This program code may be read from or written to one or more computer program products. Program code 1110 may be compressed, for example, in a suitable form.

[0108] In summary, the present application provides a data processing method, device and electronic device. In this method, when the data to be loaded is determined, the data to be loaded can be divided into blocks to obtain multiple blocks, wherein the length of the block is determined based on the remaining capacity of the first storage area; then, the multiple blocks can be loaded into the first storage area for training the model to be trained, wherein the number of blocks loaded into the first storage area each time is determined by the first number, so that through the above method, in the process of dividing the data to be loaded into blocks, the length of each block can be determined according to the remaining capacity of the first storage area, and the number of blocks loaded into the first storage area each time is relatively fixed (for example, the first number), thereby realizing that the amount of data loaded into the first storage area each time during the data loading process can be determined according to the remaining capacity of the first storage area, so that the model training process can better adapt to the storage resources in the training device, so as to facilitate more flexible use of the storage resources of the training device.

[0109] As a way, the data processing method provided in the embodiment of the present application can be further improved based on the QLoRA (Quantized Low-Rank Adaptation) method. For example, the data processing method provided in the embodiment of the present application can be understood as the DQLoRA (Dynamic Quantized Low-Rank Adaptation) method. The QLoRA method is first explained below. Among them, the QLoRA method can be regarded as an improvement based on the LoRA method, and the core improvements are as follows: (1) 4-bit NormalFloat (NF4) quantization The principle is to assume that the model weights (parameters) follow a zero-mean normal distribution and quantize the pre-trained model weights to 4-bit format while maintaining the same accuracy as 16-bit fine-tuning. The specific steps include quantile quantization, data type normalization, and data type normalization.

[0110] Among them, quantile quantification includes: according to the standard normal distribution The cumulative distribution function (CDF) of Quantiles, defining k-bit quantized data types. Quantiles The calculation formula is as follows: in is the quantile function of the standard normal distribution.

[0111] Normalization of data types includes mapping quantile quantized data type values ​​to the range of [-1, 1] to facilitate subsequent unified processing.

[0112] Data type normalization includes: converting the input weight tensor Normalize to [-1,1] and then discretize by quantile.

[0113] Dynamic reparameterization is used to compensate for quantization errors, and the accuracy loss is controlled to <1%.

[0114] The QLoRA method further compresses storage space by performing secondary quantization on the quantization constant itself. The secondary quantization includes: First quantization: weight Block quantization is performed in 4-bit NF4 format to generate the first quantization constant. .

[0115] Second quantification: Treated as new input, it is quantized into 8-bit format again to generate the second-level quantization constant and 8-bit quantization results .

[0116] (3) Paging Optimizer Application: The paging optimizer is one of QLoRA's core innovations, designed to solve the problem of training interruption caused by memory peak overflow (OOM) during large model fine-tuning. Its design is inspired by the virtual memory paging mechanism of computer operating systems. By dynamically scheduling memory resources between the GPU and CPU, it ensures that training can continue even when the memory is insufficient. Its core mechanisms include the following three points (a), (b), and (c): (a) Abstracts GPU memory and CPU memory into a unified address space. When GPU memory is insufficient, the system automatically migrates some data (such as optimizer state) to CPU memory and loads it back to the GPU on demand, eliminating the need for manual management by developers.

[0117] (b) During training, gradient checkpointing periodically saves intermediate activation values ​​to support backpropagation, resulting in transient peaks in memory usage. The paging optimizer mitigates this issue through memory pre-allocation and overflow page processing. Memory pre-allocation involves reserving a base memory pool for optimizer states (such as Adam's momentum and variance). Overflow page processing involves migrating some optimizer state pages to CPU memory when memory usage approaches a threshold.

[0118] (c) Use CUDA streams to implement asynchronous data transmission between the CPU and GPU, avoiding blocking the computing process due to data movement and ensuring that the training throughput does not decrease.

[0119] In the method provided in this application, in response to the fact that the fine-tuning scheme consumes too much video memory, a dynamic block loading mechanism and a dynamic activation cache mechanism are introduced to optimize the phenomenon that the original parameters of the large model occupy a large amount of video memory, and the activation value also occupies a large amount of video memory. And in response to the phenomenon that the rank of the low-rank matrix in LoRA and QLoRA is fixed and cannot be adapted to a variety of tasks, a dynamic rank adjustment mechanism is introduced to adapt to learning a variety of different tasks. In response to the phenomenon that the training speed is not fast enough, a hot restart and cosine annealing mechanism (Stochastic Gradient Descent with Warm Restarts, SGDR) is introduced to accelerate the training iteration process.

[0120] Using the Llama3-70B model, fine-tuning the MNLI and SST-2 tasks with different parameters, the memory consumption and training speed are as follows: Figure 11 As shown in Figure W: It can be seen that the DQLoRA method (the method provided in the embodiment of the present application) is significantly better than the LoRA method and the QLoRA method in terms of video memory usage, and fine-tuning can be completed using a single consumer-grade graphics card such as Tesla V100-SXM2-32GB. Because of the conversion from multi-card training to single-card training and the introduction of SGDR, the speed is also optimized to a certain extent, but the improvement is not very high. The reason is that the dynamic block loading mechanism, dynamic activation cache mechanism, and dynamic rank adjustment mechanism will increase the training time.

[0121] Therefore, based on the above content, the main innovations of the embodiments of the present application can be summarized as follows: (1) Regarding the large-scale model original model parameters, when fine-tuning, the original model parameters occupy a large amount of video memory. The dynamic block loading mechanism is introduced to effectively reduce the large video memory usage of the original model parameters through dynamic capacity adaptation and layer-by-layer progressive loading. Experimental observations generally reduce the video memory usage to around 20%~41%.

[0122] (2) To address the problem of activation values ​​occupying a large amount of video memory during the forward and backward propagation of large task fine-tuning, a dynamic activation cache mechanism is introduced, which can only retain about 4kb of data loaded into the video memory for a long time. This greatly alleviates the problem of excessive video memory usage during large task fine-tuning.

[0123] (3) In order to address the problem that the rank of low-rank matrices in related methods cannot be flexibly adapted to multiple tasks, a dynamic rank adjustment mechanism is introduced to identify important features through SVD and dynamic importance scoring, and dynamically adjust the rank.

[0124] (4) To address the problem of insufficient training speed, the SGDR mechanism is introduced to improve the speed of fine-tuning training.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical feature diagrams therein. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Dividing the data to be loaded into blocks to obtain a plurality of blocks, wherein the length of the blocks is determined based on the remaining capacity of the first storage area; The multiple blocks are loaded into the first storage area for training the to-be-trained model, wherein the number of blocks loaded into the first storage area each time is a first number.

2. The method according to claim 1, characterized in that When the remaining capacity of the first storage area is greater than a preset capacity threshold, configuring the length of the block to be a first length; When the remaining capacity of the first storage area is less than the preset capacity threshold, the length of the block is configured to be a second length, and the first length is greater than the second length.

3. The method according to claim 1, characterized in that The step of dividing the data to be loaded into blocks to obtain a plurality of blocks further includes: Dividing a dataset for training into a plurality of parts, wherein the dataset includes a plurality of documents and each of the parts includes at least one document; loading metadata of each of the plurality of documents into a first storage area; The data to be loaded is determined from the multiple parts according to current training requirements.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In the process of training the model to be trained, first data in the activation value of the model to be trained is stored in the first storage area, and second data in the activation value is stored in the second storage area, The access frequency of the first data is higher than the access frequency of the second data, and the data reading speed of the first storage area is faster than the data reading speed of the second storage area.

5. The method according to any one of claims 1 to 3, characterized in that: The loading the multiple blocks into the first storage area for training the model to be trained includes: Loading the multiple blocks into the first storage area for performing parameter adjustment on the low-rank matrix corresponding to the model to be trained, so as to train the model to be trained using the low-rank matrix after the parameter adjustment; The rank of the low-rank matrix is ​​determined by the importance score of the low-rank matrix.

6. The method according to claim 5, characterized in that The method further comprises: Decomposing the low-rank matrix into a plurality of matrices, wherein the plurality of matrices include a first orthogonal matrix, a second orthogonal matrix, and a diagonal singular value matrix; The importance score of the low-rank matrix is ​​obtained by the importance score of each feature in the multiple matrices, wherein the importance score of each feature is obtained by the feature sensitivity and feature uncertainty of each feature.

7. The method according to claim 5, characterized in that The rank of the low-rank matrix is ​​determined by the importance score of the low-rank matrix and the task for which the model is to be trained.

8. The method according to any one of claims 1 to 3, characterized in that: During the training of the model to be trained, the learning rate of the model to be trained is adjusted by the cosine curvature.

9. An electronic device, characterized in that: including a processor and a memory; One or more programs are stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; and the computer program product is used to implement the method according to any one of claims 1 to 8.

Images