A data processing method, a data processing device, an electronic device, a storage medium and a product

Abstract

This application provides a data processing method, a data processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product; the method includes: obtaining a first model state corresponding to a first checkpoint; the first model state is the model state output by a streaming processing engine after calling a large model to input first task processing data and performing the Nth model training; obtaining second task processing data of the streaming processing engine, where N is a positive integer; inputting the second task processing data and the first model state into a large model for training, and obtaining a second model state corresponding to a second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training on the large model.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a data processing method, a data processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale models have been widely used in many fields. Large models typically have massive numbers of parameters, and the training process requires processing large amounts of data, consuming significant computing resources and time. Especially in practical applications, models often need to be trained multiple times based on new tasks or new data to adapt to constantly changing needs.

[0003] In related technologies, when faced with multiple model training tasks, it is still necessary to reload and execute the complete training process, resulting in repeated training processes that lead to a waste of resources and time. Summary of the Invention

[0004] This application provides a data processing method, a data processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a data processing method, the method comprising: Obtain the first model state corresponding to the first checkpoint; the first model state is the model state output by the streaming engine after calling the large model input first task data to perform the Nth model training; Obtain the second task processing data of the streaming processing engine, where N is a positive integer; The second task processing data and the first model state are input into the large model for training to obtain the second model state corresponding to the second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training for the large model.

[0006] This application provides a data processing apparatus, including: The acquisition unit is used to acquire the first model state corresponding to the first checkpoint; the first model state is the model state output by the streaming processing engine after calling the large model input first task processing data to perform the Nth model training; The acquisition unit is used to acquire the second task processing data of the streaming processing engine, where N is a positive integer; The processing unit is used to input the second task processing data and the first model state into the large model for training, and obtain the second model state corresponding to the second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training for the large model.

[0007] This application provides an electronic device, which includes a communication interface and a processor, wherein... The communication interface is used to obtain the first model state corresponding to the first checkpoint; the first model state is the model state output by the streaming engine after calling the large model input first task processing data to perform the Nth model training; and to obtain the second task processing data of the streaming engine, where N is a positive integer; The processor is used to input the second task processing data and the first model state into the large model for training, and obtain the second model state corresponding to the second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training for the large model.

[0008] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in this application when executed by a processor.

[0009] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method provided in this application.

[0010] The embodiments of this application have the following beneficial effects: by obtaining the first model state corresponding to the first checkpoint; the first model state is the model state output by the streaming engine after calling the large model to input the first task processing data and performing the Nth model training; obtaining the second task processing data of the streaming engine, where N is a positive integer; inputting the second task processing data and the first model state into the large model for training, and obtaining the second model state corresponding to the second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training for the large model; it can be seen that by recording the current model state as a checkpoint during each model training and performing incremental learning based on the state of the previous checkpoint in the next training, this application avoids training the entire model from scratch, thereby significantly reducing the consumption of computational resources and time costs in the repeated training process. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of this application; Figure 2This is a schematic diagram illustrating the process of starting Flink tasks for model training at different points in time, as provided in the embodiments of this application. Figure 3 This is a schematic diagram illustrating the usage flow of Flink checkpoint at the start of model training and during incremental learning, as provided in the embodiments of this application. Figure 4 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0015] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0016] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0017] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0018] Figure 1 This is a first flowchart illustrating the data processing method provided in the embodiments of this application. The following will be combined with... Figure 1 The steps shown will be explained. It should be noted that... Figure 1 The method described uses an electronic device as the execution subject as an example. Figure 1 As shown, the method includes the following steps 101 to 103. Step 101: Obtain the first model state corresponding to the first checkpoint; the first model state is the model state output by the streaming engine after calling the large model input first task processing data to perform the Nth model training.

[0019] In this embodiment, the first checkpoint (checkpoint1) refers to a snapshot file generated during the execution of a streaming engine (such as Flink) task, which persistently stores the current task's state information according to a set interval or event triggering mechanism. The first checkpoint contains the model's state information after the Nth training iteration, such as key variables like model parameters and optimizer state. This state information is stored in a designated state backend, such as a distributed storage system like the Hadoop Distributed File System (HDFS) or Amazon SimpleStorage Service (S3), for subsequent task recovery.

[0020] The first model state refers to the set of key variables within the large model after the Nth training iteration, including but not limited to weights and biases in the neural network, and momentum terms and squared gradients in the optimizer. These states determine the current learning ability and predictive performance of the large model and are used to continue training on new data during incremental learning, thereby avoiding the need to reinitialize the entire large model.

[0021] In this embodiment, the first checkpoint serves to quickly save and restore the state of the large model, which in turn carries the training results of the large model within a certain period of time.

[0022] For example, suppose we are currently training a large image recognition model. After the Nth iteration, the model has acquired a certain level of recognition capability. At this point, the Flink task will generate a checkpoint file, saving the parameters and optimizer state of the current large model for use in subsequent tasks.

[0023] For example, suppose the large model is a text classification model in natural language processing. After multiple iterations, the large model can understand different semantic structures more accurately, and the checkpoint at this time also records the state of the large model at this stage.

[0024] In practical applications, after a Flink task completes a full training cycle, it automatically generates a checkpoint file and writes the current state of the large model into it. This not only preserves the complete state of the large model at a given moment but also allows subsequent tasks to continue training based on the existing large model, significantly reducing the time and computational resources required for repeated training. Furthermore, since the large model state is generated based on the training results of the previous stage, subsequent training processes are more coherent and stable.

[0025] Step 102: Obtain the second task processing data of the streaming engine, where N is a positive integer.

[0026] In this embodiment, the second task processing data refers to a new batch of data to be processed based on the first model state. This data typically comes from real-time or batch data sources, such as user behavior logs, sensor data, and text data, depending on the application scenario.

[0027] In this embodiment, the streaming engine is responsible for reading data from external data sources and distributing the data to the large model for training according to a preset method. The data processed in the second task has a similar format and structure to the data processed in the first task, but the content may differ. The data processed in the second task represents new input samples.

[0028] For example, in image recognition scenarios, the second task processing data might be a new set of images used to further improve the generalization ability of the large model. In natural language processing scenarios, the second task processing data might be a new text corpus used to expand the language understanding and expressive capabilities of the large model. In recommendation scenarios, the second task processing data might be the user's latest click behavior data used to update the recommendation strategy of the large model.

[0029] In practical applications, the streaming engine continuously monitors changes in the data source and automatically loads and distributes new data when it detects it. This enables dynamic data loading and real-time updates of large models, thereby improving response speed and adaptability. Furthermore, since the large model state is generated from the previous training task, the addition of new data does not destroy the already completed training results; instead, it allows for further optimization and improvement based on these results.

[0030] Step 103: Input the second task processing data and the first model state into the large model for training to obtain the second model state corresponding to the second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training for the large model.

[0031] In this embodiment, the second checkpoint (checkpoint2) refers to a snapshot file generated again after the current training is completed, based on the set interval or event triggering mechanism, to save the state information after the N+1th model training.

[0032] The second model state refers to the set of key variables within the large model at the end of the (N+1)th training iteration, including the large model parameters and the optimizer state. Compared to the first model state, the second model state reflects how the large model has changed under the influence of new data; for example, these parameters may have been updated significantly, or the optimizer state may have changed markedly.

[0033] There is also a direct correlation between the second checkpoint and the second model state, which is recorded through the second checkpoint. In other words, the role of the second checkpoint is to quickly save the current large model state, providing a basis for incremental learning in subsequent tasks.

[0034] For example, during the training of a large image recognition model, if the data processed by the second task contains images with more complex backgrounds, the large model may adjust the weights of its feature extraction layer in the N+1th training iteration to better adapt to the complex environment.

[0035] For example, in a large text classification model, if the data processed by the second task contains more polysemous words or ambiguous sentence structures, the large model may enhance its contextual modeling capabilities to improve classification accuracy.

[0036] In practical applications, the process of inputting the data from the second task and the state of the first model into the large model for training is essentially an incremental learning process. By using the model state generated from the previous training as the initial state, the parameters of the large model can be avoided, thereby significantly reducing training time and resource consumption. Furthermore, since each training iteration is based on the results of the previous iteration, the update process of the large model is more stable and controllable, which helps to improve the overall performance and generalization ability of the large model.

[0037] In summary, the data processing method provided in this application combines the Flink checkpoint mechanism with the large model training process to achieve efficient recording and recovery of the model state. This allows subsequent training tasks to perform incremental updates based on the existing model state, thereby greatly improving the efficiency and resource utilization of model training. By introducing the checkpoint mechanism, not only can the model state be quickly recovered, but the resource consumption and time cost during training can also be effectively reduced, improving the overall performance of model training.

[0038] The data processing method provided in this application obtains the first model state corresponding to the first checkpoint; the first model state is the model state output by the streaming processing engine after calling the large model to input the first task processing data and performing the Nth model training; obtains the second task processing data of the streaming processing engine, where N is a positive integer; inputs the second task processing data and the first model state into the large model for training, and obtains the second model state corresponding to the second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training for the large model; it can be seen that this application avoids training the entire model from scratch by recording the current model state as a checkpoint during each model training and performing incremental learning based on the state of the previous checkpoint in the next training, thereby significantly reducing the consumption of computational resources and time costs in the repeated training process.

[0039] In some embodiments, before step 101 obtains the first model state corresponding to the first checkpoint, the above method further includes: Step 201: Define the initial model state of the large model using the streaming engine.

[0040] In this embodiment, the streaming engine is a computing framework for real-time processing of large-scale continuous data streams, capable of performing data cleaning, transformation, and aggregation operations without interrupting the data flow. In this application, the streaming engine is used to initialize the state of a large model, i.e., to set the initial parameters and structure of the model, providing a foundation for the subsequent training process.

[0041] Streaming engines support various computational tasks, such as windowing, state management, and event-driven processing. They can define and store the initial state of large models in a structured manner, ensuring that subsequent incremental learning can proceed consistently. For example, Flink is a streaming engine with powerful state management and fault tolerance capabilities, making it suitable for large model training scenarios.

[0042] In practical applications, the streaming engine loads the model architecture according to the configuration and initializes model parameters, such as weight matrices and bias terms. These initial parameters constitute the initial model state of the large model, serving as the starting point for training. The streaming engine can also be integrated with external storage systems (such as HDFS) to recover the model state in the event of a failure.

[0043] In this embodiment, defining the initial model state of a large model using a streaming processing engine ensures the repeatability and consistency of model training, thereby improving the efficiency and stability of model training. This method of defining the initial model state of a large model using a streaming processing engine avoids redefining the model structure for each training iteration, thus saving resources and time.

[0044] Step 202: Obtain the first task processing data from the streaming engine.

[0045] In this embodiment, task processing data refers to the data input generated or received by the streaming engine when executing the first task. Task processing data typically originates from real-time data streams from business systems, such as user behavior logs, sensor data, and transaction records. After preprocessing, the task processing data becomes input samples for training large models, used to update model parameters and optimize model performance.

[0046] In practical applications, task processing data may include raw feature vectors, label information, and metadata. Streaming engines batch process the task processing data at certain time intervals or data volume thresholds and then pass the processed data to a large model for training. Furthermore, task processing data may also contain historical data fragments to build contextual information and improve the generalization ability of the large model.

[0047] Here, by acquiring task processing data from the streaming engine, we can ensure that the data used during the training of large models is high-quality and real-time, thereby improving the model's learning efficiency and prediction accuracy. Acquiring task processing data from the streaming engine also makes the model more adaptable to new input data.

[0048] Step 203: Input the data processed by the first task and the initial model state into the large model for training, and obtain the first model state output by the large model.

[0049] In this embodiment, during training, the large model performs feature extraction, model inference, and parameter updates on the input data, ultimately outputting a new model state, denoted as the first model state. The first model state contains the updated model parameters and intermediate variables, which can be used for subsequent incremental learning.

[0050] In actual training, the model can use optimization algorithms such as gradient descent and Adaptive Moment Estimation (Adam) to adjust its parameters. Simultaneously, the training process is also affected by hyperparameters such as learning rate, batch size, and number of iterations. To ensure model stability and convergence, regularization mechanisms and early stopping strategies are typically incorporated during training.

[0051] By training the model using task-processed data combined with the initial model state, a more accurate and robust large model can be obtained. This training process enables the model to have higher predictive and generalization abilities when faced with new data inputs. Since model training is performed based on the initial model state, this method can significantly reduce training time and resource consumption.

[0052] In this embodiment, a streaming engine is used to define the initial model state of a large model, and training is performed using task-processed data, thereby achieving high efficiency and repeatability in model training. This method can reduce the resource and time costs consumed by repeated training of large models, thus improving the model's practicality and scalability.

[0053] In some embodiments, after step 203 inputs the first task processing data and the initial model state into the large model for training and obtains the first model state output by the large model, the above scheme further includes: Store the state of the first model in a distributed file system; At this point, obtaining the first model state corresponding to the first checkpoint includes: obtaining the first model state from the distributed file system.

[0054] In this embodiment, the distributed file system is a file system capable of storing and managing large-scale data across multiple servers, possessing high availability, scalability, and fault tolerance. The distributed file system used in this application can be HDFS, S3, or other similar file systems. The distributed file system supports persistent storage of model state in the form of files, ensuring that the model state is not lost even in the event of hardware failure or system restart.

[0055] In practical applications, storing the first model state in a distributed file system can include serializing the first model state into a byte stream and writing the byte stream to a file at a specified path. This process is automatically triggered by the Checkpoint mechanism at the end of the Flink task. By storing the first model state in a distributed file system, the first model state can be preserved long-term and can be quickly restored in subsequent incremental learning tasks.

[0056] This application achieves reliable persistence of the model state by storing it in a distributed file system. Storing the model state in a distributed file system avoids the resource waste associated with retraining the entire model each time, thereby reducing the time cost and computational resource consumption of model training, and ultimately improving overall model training efficiency and enhancing system stability and reliability.

[0057] In this embodiment, a checkpoint is a mechanism in Flink used to record the execution state of a task. During task execution, checkpoints periodically save the current state information to persistent storage. In this application, each checkpoint contains a complete set of model state variables, such as model parameters and optimizer state. In the next training task, these model state variables can be loaded to achieve incremental learning.

[0058] The process of retrieving the first model state from the distributed file system occurs when starting a new Flink task. This involves configuring the new task to use FsStateBackend (the file system state backend) and specifying the path where the last checkpoint was stored. Flink reads the corresponding model state file based on this path and restores it to the task's state management module. This process is crucial for incremental learning, as correctly restoring the previous training state is essential for training new data on top of that state.

[0059] By retrieving the first model state from a distributed file system, the results of the previous model training can be quickly recovered. Using the first model state stored in the distributed file system for current model training avoids repeatedly training some parameters of an existing model, thereby reducing computational resource consumption, accelerating the current model training process, and improving the flexibility and responsiveness of the model iteration process.

[0060] In some embodiments, before obtaining the first model state corresponding to the first checkpoint in step 101, the above method further includes setting a minimum interval time between the first checkpoint and the second checkpoint.

[0061] In this embodiment, the minimum interval time refers to the shortest time interval that must be maintained between the first checkpoint and the second checkpoint. The user-configurable minimum interval time setting is to prevent checkpoints from being generated too frequently, thereby avoiding resource waste and performance degradation. In Flink, the minimum interval time can be set through the configuration parameter `minPauseBetweenCheckpoints`. For example, the user can set the minimum interval time to 5 seconds, 10 seconds, or longer, depending on the complexity of the training task and resource availability.

[0062] During model training, if the interval between the first and second checkpoints is too short, a large amount of duplicate data will be saved, increasing the storage burden; conversely, if the interval is too long, it may reduce the timeliness of model recovery. Therefore, setting a reasonable minimum interval is crucial for balancing resource usage and improving model recovery efficiency.

[0063] In practical applications, taking a large model training task as an example, suppose a checkpoint is generated every 5 minutes during a training task. If the large model training task has just completed a checkpoint, the next checkpoint cannot be triggered again until at least 5 minutes later, regardless of whether other conditions are met (such as data processing completion). Setting a minimum interval between the first and second checkpoints helps control the frequency of checkpoint generation, making the checkpoint generation process more stable and controllable.

[0064] In this embodiment of the application, by setting a minimum interval between the first checkpoint and the second checkpoint, the operation can avoid generating checkpoints too frequently, thereby reducing the consumption of computing and storage resources during model training. This can optimize the resource utilization efficiency during model training, thereby improving the overall training efficiency and reducing operating costs.

[0065] In some embodiments, before obtaining the first model state corresponding to the first checkpoint in step 101, the above method further includes setting the timeout time for the first checkpoint and the second checkpoint.

[0066] In this embodiment, setting the timeout for the first and second checkpoints refers to configuring a time period within the Flink framework. If the processing of the first or second checkpoint is not completed within the configured time period, it is determined that the first or second checkpoint has expired and is automatically cleared. This mechanism of setting the timeout for the first and second checkpoints controls the validity and lifecycle of checkpoint data, preventing excessive storage resource consumption due to prolonged periods without updates. By setting the timeout appropriately, operational efficiency can be improved while ensuring data reliability.

[0067] The timeout setting is usually based on a comprehensive consideration of factors such as task execution frequency, data volume, and load conditions. For example, in high-concurrency, large-data-volume training tasks, the timeout can be appropriately extended to ensure that the first and second checkpoints can be stably recorded; while in low-load or high-real-time-requirement scenarios, the timeout can be shortened to speed up state cleanup and reduce the accumulation of invalid states.

[0068] Furthermore, setting a reasonable timeout helps prevent checkpoint accumulation caused by network interruptions, compute node failures, and other issues. Apache Flink determines whether to retain the current checkpoint record based on the configured timeout, thereby enabling dynamic management of the state information of the first machine learning model running in the stream processing task.

[0069] In practical applications, users can specify the timeout for each checkpoint through Flink's API or configuration files. For example, adding the `state.checkpoints.dir` and `execution.checkpointing.timeout` parameters to the Flink configuration specifies the checkpoint storage path and timeout, respectively. This allows for automatic management of the checkpoint lifecycle based on a preset time strategy.

[0070] In this embodiment, timeout periods are set for the first and second checkpoints. Setting these timeout periods effectively manages the lifecycle of the checkpoints and avoids the accumulation of invalid states. This approach improves resource utilization and operational efficiency, and better supports the incremental learning needs during the training of the first model.

[0071] In some embodiments, the model state includes one or more of the following: model parameters, optimizer state.

[0072] In this embodiment, model parameters refer to the set of variables such as weight matrices and bias vectors used during the training of a deep learning model. Model parameters determine the model's predictive ability and are continuously updated by optimization algorithms such as gradient descent in each iteration. For example, in a neural network, the weights and biases of each layer are part of the model parameters. Model parameters determine how input data is processed and mapped to the output. In this application, model parameters, as key state information during the training process of the deep learning model, are recorded in the Flink checkpoint for incremental learning in subsequent tasks.

[0073] In this embodiment, the optimizer state refers to the set of additional variables maintained by the optimization algorithm during model training. Optional optimizers, such as Adam, maintain state information such as momentum and second moment estimate. This state information is used to adjust the learning rate, prevent oscillations, and accelerate convergence. The optimizer state directly affects the effectiveness and efficiency of model training; therefore, including the optimizer state in the scope of model state storage helps to more comprehensively restore the consistency of the training process.

[0074] Therefore, saving the model parameters and optimizer state together in the checkpoint ensures that the system can quickly revert to the same state as the previous training in subsequent incremental learning tasks. This avoids the waste of resources and time caused by re-initialization. Saving the model parameters and optimizer state together in the checkpoint also improves the reusability and traceability of model training, facilitating debugging and version management.

[0075] In practical applications, the synchronized saving of model parameters and optimizer states not only ensures the continuity of the training process but also provides a guarantee for fault tolerance and recovery in distributed training environments. Through unified management of model parameters and optimizer states, it is possible to quickly recover to the most recent stable state after a task interruption, thereby improving overall training efficiency and robustness.

[0076] In summary, by recording model parameters and optimizer states during model training, the complete preservation of the model state is achieved. This allows for rapid recovery of the model state in subsequent incremental learning tasks, reducing the time cost of repeated training and significantly lowering the computational resources and energy consumption required for training large models.

[0077] In a feasible scenario, this application proposes a method for incremental learning in model training based on Flink checkpointing, which greatly optimizes the resource and time consumption during repeated training of large models. To more intuitively demonstrate the implementation of this application, Figure 2This is a schematic diagram illustrating the process of starting a Flink task for model training at different points in time, as provided in the embodiments of this application.

[0078] Figure 2 This presentation shows the overall process of model training in Flink tasks starting at three different points in time. All Flink tasks have checkpointing enabled, periodically recording the execution status of model training within each task.

[0079] (1) The initial state timeline1 is the time point when Flink job1 first performs checkpoint recording. When Flink job1 ends, the checkpoint recording stops on timeline2.

[0080] (2) Incremental learning state: Flink job2 uses the checkpoint record of Flink job1 on timeline2 to resume the task and start a new incremental learning. When Flink job2 ends, the checkpoint record stops on timeline3.

[0081] (3) In the incremental learning state, Flink job3 uses the checkpoint record of Flink job2 on timeline3 to resume the task and start a new incremental learning. When Flink job3 ends, the checkpoint record stops on timeline4.

[0082] Therefore, apart from the first training task, for each subsequent training task, we only need to start from the training state stored in the checkpoint record of the previous task and continue training on the new data.

[0083] Figure 3 This is a schematic diagram illustrating the usage flow of Flink checkpoint at the start of model training and during incremental learning, as provided in the embodiments of this application.

[0084] (1) The process of using Flink checkpoints during initialization: (1.1) Initialize the model: Determine the state variables, such as model parameters and optimizer states, which need to be recorded during the training of a large model. Define the states in the Flink program using Flink's state management APIs, such as ValueState and ListState, to define the state variables that need to be recorded.

[0085] (1.2) Enable and configure the Flink checkpoint mechanism: Set the checkpoint mechanism to enabled, and also set the appropriate trigger time for checkpoints, the minimum interval between two adjacent checkpoints, and the timeout period.

[0086] Set the checkpoint not to be deleted.

[0087] Configure the storage and storage directory for checkpoints; here, HDFS is selected. For example, hdfs: / / localhost:8020 / flink / checkpoints.

[0088] (1.3) Import model training data into Flink task and start training: Start the Flink task to begin processing the model training data. Update the state in the processing function, and update the state variables according to the training progress of the large model. For example, update the state of the model parameters after each iteration.

[0089] (1.4) The model training phase ends: Control the amount of data used for training a Flink task, and then terminate the current task.

[0090] (1.5) Record the checkpoint (denoted as CP1): After a Flink task completes, model training is finished. The latest checkpoint stores the model states that needed to be recorded during this training session, denoted as CP1. Flink automatically saves the model's training state variables in the designated storage. These will be used for incremental learning in the next training task.

[0091] (2) The process of using Flink checkpoint during incremental learning: (2.1) Enable and configure the Flink checkpoint mechanism: During incremental learning, the checkpoint mechanism is also enabled, and the process is the same as "step (1.2) of the Flink checkpoint usage process during initialization".

[0092] (2.2) Resuming a new Flink task using CP1: Enable StateBackend in the Flink configuration.

[0093] For example, `env.setStateBackend(new FsStateBackend("hdfs: / / localhost:8020 / flink / checkpoints"))` will cause Flink to restore the state from the most recent checkpoint (CP1).

[0094] CP1 is the most recent checkpoint record from the last Flink model training. Flink automatically reads the state variables from the checkpoint and restores them to the processing function.

[0095] (2.3) Input new data and continue training the Flink task: In a new Flink task, new data is input, and training can continue based on the previous training results. The training state is updated in the processing function, and the state variables are updated according to the training progress of the large model to achieve incremental learning.

[0096] (2.4) The model training phase ends: Control the amount of data trained by the Flink task during this incremental learning phase, and then terminate the current task.

[0097] (2.5) Record the checkpoint (denoted as CP2): After a Flink task completes, model training is finished. The latest checkpoint stores the model states that needed to be recorded during this training session, denoted as CP2. Flink automatically saves the model's training state variables in the designated storage. These will be used for incremental learning in the next training task.

[0098] In summary, by using Flink checkpoint to record the training state after each model training iteration, only incremental learning is needed during repeated training, significantly reducing time and resource consumption. This application achieves the recording and rapid recovery of model training state through the Flink checkpoint mechanism. This method significantly optimizes the time and resources consumed during repeated model training and facilitates later model reuse and incremental learning.

[0099] This application also provides a data processing apparatus for use in electronic devices, such as... Figure 4 As shown, the data processing apparatus includes: The acquisition unit 401 is used to acquire the first model state corresponding to the first checkpoint; the first model state is the model state output by the streaming processing engine after calling the large model input first task processing data to perform the Nth model training; Acquisition unit 401 is used to acquire the second task processing data of the streaming engine, where N is a positive integer; The processing unit 402 is used to input the second task processing data and the first model state into the large model for training, and obtain the second model state corresponding to the second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training for the large model.

[0100] In some embodiments, processing unit 402 is used to define the initial model state of a large model through a streaming processing engine; Acquisition unit 401 is used to acquire the first task processing data of the streaming processing engine; The processing unit 402 is used to input the first task processing data and the initial model state into the large model for training, and obtain the first model state output by the large model.

[0101] In some embodiments, processing unit 402 is used to store the first model state in a distributed file system; The acquisition unit 401 is used to acquire the first model state corresponding to the first checkpoint by performing the following steps: acquiring the first model state from the distributed file system.

[0102] In some embodiments, the processing unit 402 is used to set the minimum interval time between the first checkpoint and the second checkpoint.

[0103] In some embodiments, the processing unit 402 is configured to set the timeout periods for the first checkpoint and the second checkpoint. In some embodiments, the model state includes one or more of the following: model parameters, optimizer state.

[0104] This application also provides an electronic device, such as... Figure 5 As shown, the electronic device 500 includes: a communication interface 501 and a processor 502; wherein, The communication interface 501 enables information exchange with other devices (such as external data sources and distributed file systems); The processor 502 is connected to the communication interface 501 to enable information interaction with other devices and to execute the methods provided by one or more technical solutions on the electronic device side when running a computer program. Memory 503 stores computer programs that can run on processor 502.

[0105] Communication interface 501 is used to obtain the first model state corresponding to the first checkpoint; the first model state is the model state output by the streaming engine after calling the large model input first task processing data to execute the Nth model training; and to obtain the second task processing data of the streaming engine, where N is a positive integer; The processor 502 is used to input the second task processing data and the first model state into the large model for training, and obtain the second model state corresponding to the second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training for the large model.

[0106] In some embodiments, processor 502 is used to define the initial model state of a large model via a streaming engine; Communication interface 501 is used to obtain the first task processing data of the streaming engine; Processor 502 is used to input the first task processing data and the initial model state into the large model for training, and obtain the first model state output by the large model.

[0107] In some embodiments, processor 502 is used to store the first model state to a distributed file system; Communication interface 501 is used to obtain the first model state from the distributed file system when obtaining the first model state corresponding to the first checkpoint.

[0108] In some embodiments, processor 502 is configured to set the minimum interval time between the first checkpoint and the second checkpoint.

[0109] In some embodiments, processor 502 is configured to set timeout times for the first checkpoint and the second checkpoint.

[0110] In some embodiments, the model state includes one or more of the following: model parameters, optimizer state.

[0111] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0112] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in this application.

[0113] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the data processing method provided in this application. For example, ... Figure 1 The data processing method is shown.

[0114] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0115] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0116] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0117] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0118] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the first model state corresponding to the first checkpoint; the first model state is the model state output by the streaming engine after calling the large model input first task data to perform the Nth model training; Obtain the second task processing data of the streaming processing engine, where N is a positive integer; The second task processing data and the first model state are input into the large model for training to obtain the second model state corresponding to the second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training for the large model.

2. The method according to claim 1, characterized in that, When N is 1, before obtaining the first model state corresponding to the first checkpoint, the process includes: The initial model state of the large model is defined through a streaming engine; Obtain the first task processing data of the streaming engine; The first task processing data and the initial model state are input into the large model for training, and the first model state output by the large model is obtained.

3. The method according to claim 2, characterized in that, After inputting the first task processing data and the initial model state into the large model for training, and obtaining the first model state output by the large model, the method further includes: Store the state of the first model in a distributed file system; The step of obtaining the first model state corresponding to the first checkpoint includes: Obtain the state of the first model from the distributed file system.

4. The method according to claim 1, characterized in that, Before obtaining the first model state corresponding to the first checkpoint, the method further includes: Set the minimum interval between the first checkpoint and the second checkpoint.

5. The method according to claim 1, characterized in that, Before obtaining the first model state corresponding to the first checkpoint, the method further includes: Set the timeout time for the first checkpoint and the second checkpoint.

6. The method according to any one of claims 1 to 5, characterized in that, The model state includes one or more of the following: model parameters, optimizer state.

7. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire the first model state corresponding to the first checkpoint; The first model state is the model state output by the streaming engine after calling the large model input first task data to perform the Nth model training; The acquisition unit is used to acquire the second task processing data of the streaming processing engine, where N is a positive integer; The processing unit is used to input the second task processing data and the first model state into the large model for training, and obtain the second model state corresponding to the second checkpoint output by the large model. The second model state is the model state output after performing the (N+1)th model training for the large model.

8. An electronic device, characterized in that, Including communication interfaces and processors, among which, The communication interface is used to obtain the first model state corresponding to the first checkpoint; the first model state is the model state output by the streaming engine after calling the large model input first task processing data to perform the Nth model training; and to obtain the second task processing data of the streaming engine, where N is a positive integer; The processor is used to input the second task processing data and the first model state into the large model for training, and obtain the second model state corresponding to the second checkpoint output by the large model; the second model state is the model state output after performing the N+1th model training for the large model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images