Model training methods, data processing methods, systems and electronic equipment

Abstract

This application provides a model training method, system, electronic device, and storage medium. The method includes: acquiring target offline data and target real-time data for training a target model from a stored training set; training the model to be trained based on the target offline data and target real-time data to obtain a trained model; subsequently, during the training of the model to be trained, if updated first real-time data for training the target model is detected in the training set, the first real-time data is identified as the target real-time data; finally, in response to the detection of updated second real-time data for training the target model in the training set, updating the trained model based on the second real-time data to obtain an updated trained model. This technical solution continuously updates the trained model upon detecting updates to real-time data, thereby enhancing the real-time performance and reliability of model training.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, a data processing method, a system, an electronic device, and a computer-readable storage medium. Background Technology

[0002] The core of data science revolves around big data and artificial intelligence technologies. With the development of artificial intelligence technology, machine learning models need to be built for various business scenarios. The establishment of different machine learning models is often accompanied by data support provided by big data.

[0003] In existing technologies, the establishment of common machine learning models is based on offline data in big data. However, for real-time generated data, specialized personnel are needed to acquire and process it before it can help machine learning models.

[0004] In the above technical solutions, there is a problem with the low efficiency of using real-time data during the construction of machine learning models, resulting in poor reliability and real-time performance of the generated machine learning models. Summary of the Invention

[0005] In view of this, this application provides a model training method, a data processing method, a system, an electronic device, and a storage medium to improve the reliability and real-time performance of model training.

[0006] The first aspect of this application provides a model training method, the method comprising:

[0007] Obtain target offline data and target real-time data for training the target model from the stored training set used for training the target model. The real-time data in the training set is obtained by acquiring data generated on the data generation device in real time.

[0008] Based on the target offline data and the target real-time data, the model to be trained is trained to obtain the trained model;

[0009] During the training process of the model to be trained, if it is detected that there is updated first real-time data in the training set used to train the target model, the first real-time data is determined as the target real-time data;

[0010] In response to the detection of updated second real-time data for training the target model in the training set, the trained model is updated and trained based on the second real-time data to obtain an updated trained model.

[0011] A second aspect of this application provides a model training system, the system comprising: a data capability module and a data training module;

[0012] The data capability module is used to obtain target offline data and target real-time data for training the target model from the stored training set used for training the target model. The real-time data in the training set is obtained by acquiring data generated on the data generation device in real time.

[0013] The data training module is used to train the model to be trained based on the target offline data and the target real-time data to obtain the trained model.

[0014] The data capability module is also used to, during the training process of the model to be trained, if it is detected that there is updated first real-time data in the training set used to train the target model, determine the first real-time data as the target real-time data.

[0015] The data training module is further configured to, in response to the detection of updated second real-time data for training the target model in the training set, update the trained model based on the second real-time data to obtain an updated trained model.

[0016] A third aspect of this application provides a data processing method, the method comprising:

[0017] Acquire data to be processed and a processing model for processing the data to be processed, wherein the processing model is trained according to the model training method described in any one of the first aspects;

[0018] The processing model is used to process the data to be processed, and the processing result is obtained.

[0019] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the model training method as described in any one of the first aspects above.

[0020] A fifth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the model training method as described in any one of the first aspects above.

[0021] The technical solution provided in this application obtains target offline data and target real-time data for training a target model from a stored training set. The training set stores offline data and real-time data for model training. The real-time data in the training set is obtained by acquiring data generated on a data generation device in real time. Based on the target offline data and target real-time data, the model to be trained is trained to obtain a trained model. Then, during the training of the model to be trained, if updated first real-time data for training the target model is detected in the training set, the first real-time data is identified as the target real-time data. Finally, in response to the detection of updated second real-time data for training the target model in the training set, the trained model is updated and trained based on the second real-time data to obtain an updated trained model. This technical solution continuously updates and trains the trained model upon detecting updates to real-time data, thereby enhancing the real-time performance and reliability of model training. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating an embodiment of the model training method provided in this application;

[0024] Figure 2 A schematic diagram of the model training system provided in the embodiments of this application;

[0025] Figure 3 A schematic flowchart of Embodiment 2 of the model training method provided in this application;

[0026] Figure 4 A diagram illustrating data synchronization for existing technologies;

[0027] Figure 5 This is a data synchronization diagram provided for an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the existing real-time data warehouse processing flow.

[0029] Figure 7 This is a schematic diagram of the existing offline data warehouse processing flow.

[0030] Figure 8 This is a schematic diagram of the integrated batch and stream data warehouse processing flow provided in the embodiments of this application;

[0031] Figure 9 A schematic diagram illustrating the principle of the data capability module provided in an embodiment of this application;

[0032] Figure 10 A flowchart illustrating the data processing method provided in an embodiment of this application;

[0033] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions of this application, the application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. However, this application can be implemented in many other ways different from those described above. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0035] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described herein. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0036] First, the technical background and technical terms involved in the embodiments of this application will be introduced and explained.

[0037] Data science is a discipline that has emerged in recent years, primarily studying how to discover regular patterns in massive amounts of data and provide insights for decision-making. Data science has many subcategories, such as machine learning, deep learning, machine / deep learning frameworks, big data frameworks, big data development, and database systems. This complex division of labor has laid the foundation for the current booming field of artificial intelligence. Data science has also spawned many professional roles, such as data scientists, algorithm engineers, and big data engineers, which have become highly sought-after professions. In some contexts, data science and artificial intelligence share similar semantics; in fact, data science encompasses a broader range of data-based research, and artificial intelligence can be seen as a subset of data science.

[0038] Gaming is one of the earliest fields to apply artificial intelligence (AI). From intelligent non-player character (NPC) robots to AI-powered character creation algorithms, gaming has become fertile ground for AI applications. Although AI has been developing for decades, the software and hardware systems surrounding it are still in their early stages of development, making the industrialization of AI particularly urgent. Here, industrialization typically refers to the formation of methodologies and the creation of an ecosystem starting from software and hardware platforms. In fact, traditional software and hardware development over the past few decades has formed a mature industrial methodology and produced many groundbreaking products, such as central processing units (CPUs), the open-source Linux operating system, Oracle databases, and Netscape Navigator. These hardware and software systems work together to build an extremely diverse range of applications, from dedicated chips and software systems for commercial aircraft to personal computers and software used by ordinary people. In recent years, building a software and hardware ecosystem around artificial intelligence has become a key focus. Starting with the rise of deep learning in artificial intelligence, various companies have successively released proprietary AI chips such as Tensor Processing Units (TPUs), software frameworks such as TensorFlow and PyTorch, a series of high-performance graphics processing units (GPUs) and their supporting software (Compute Unified Device Architecture, CUDA). These data science software and hardware systems have sprung up like mushrooms after rain, playing a significant role in popularizing AI applications and giving rise to a new software and hardware paradigm in the AI ​​era.

[0039] In existing technologies, data science revolves around big data and artificial intelligence, and existing solutions are all developed around these two ecosystems. From a software perspective, the big data field has open-source systems such as Hadoop, HDFS, Yarn, Spark, and Flink; the artificial intelligence field has open-source frameworks such as TensorFlow and PyTorch; the data analysis field has data processing standards or software toolkits such as Structured Query Language (SQL), Spark, and Flink; and the cloud-native ecosystem includes Docker and Kubernetes. From a hardware perspective, big data technologies generally run on ordinary large-scale CPU clusters, while artificial intelligence requires dedicated GPU clusters to ensure the efficiency of its vector computation and backpropagation algorithms.

[0040] Above the infrastructure lies the middleware and platform layer. Engineers develop a series of platforms and tools based on these infrastructure technologies, such as offline computing platforms, real-time computing platforms, and ad-hoc query platforms for big data development, and machine learning / deep learning platforms and containerized deployment platforms for artificial intelligence development. Furthermore, there may be platforms that focus more on the business side, such as advertising platforms and recommendation experimentation platforms.

[0041] However, few existing systems were designed from the outset with the specificities of data science in mind.

[0042] Technically, the two most important systems in data science (big data platforms and machine learning platforms) are often fragmented, lacking interoperability. For example, data accumulated in big data systems cannot be efficiently accessed in machine learning systems, causing significant inconvenience to algorithm development and iteration. Algorithm engineers are forced to learn both big data and AI platforms simultaneously and then manually code to integrate functionality. This implementation is often customized and not easily reusable. Some machine learning and big data systems achieve a certain degree of resource interoperability through open application programming interfaces (APIs), but this is often accompanied by numerous limitations. Furthermore, to deploy algorithm models to mobile hardware, algorithm engineers may need to learn model optimization and simplification, such as writing CUDA code. Admittedly, large-scale enterprises can hire professional CUDA engineers to handle this work, but for small and medium-sized teams and enterprises, this often means high costs.

[0043] From a business perspective, data science is still in its early stages, and the industry faces a multitude of real-world business challenges. Applying data science methods to practical business applications presents significant difficulties. Take the gaming industry as an example:

[0044] First, business requirements are often quite diverse. The requirements proposed by game designers often require game AI engineers to perform multi-level abstraction before they can be modeled as data science problems. Specifically, there is a high degree of uncertainty in both business and technology aspects. Some requirements, after being abstracted, may find that there are no industry-relevant models for reference, making implementation difficult and deployment challenging.

[0045] Secondly, the sheer variety of games, differing significantly in gameplay and target audience, places higher demands on algorithm engineers. A deep understanding of the game is often more crucial than the algorithm itself. While e-commerce recommendation strategies for women's clothing can be easily transferred to men's clothing, such direct transfer is rarely effective for games. Custom design is essential, requiring tailored development to the specific gameplay. Algorithms implemented in one game are difficult to quickly migrate to another. Specifically, this demands a high level of game expertise from technical personnel, requiring a deep understanding of games and the ability to grasp the core needs of game designers and transform problems into AI sub-problems. Furthermore, engineering reusability is low. AI algorithms need to be customized for game clients / servers and may even be integrated into the main gameplay loop. Consequently, the technology stack will inevitably converge with the game development technology stack, which can vary greatly between games, making it difficult to quickly replicate algorithms implemented in one game in another.

[0046] Third, the evaluation methods for game performance differ from those used in e-commerce scenarios, such as click-through rate, repurchase rate, and Gross Merchandise Volume (GMV). The effects of combining games and AI need to be considered from multiple dimensions, such as fun and visual appeal, and also from a long-term perspective. A new gameplay feature might have a positive effect on the game in the short term, but could be negative in the long run. Specifically, applying industry-standard performance evaluation methods like A / B testing to games can face many challenges, such as whether the service will affect the core gameplay; the evaluation cycle might take months; and researchers might not receive timely feedback and be able to iterate quickly on the service.

[0047] Fourth, traditional methodologies widely studied in internet scenarios (such as search and recommendation) have limited applicability in gaming. Games require more innovative methods and technologies (even those still in the research stage in academia), such as new technologies in computer vision and audio-visual fields, like style transfer and intelligent creation generation. Applying data science technologies to the gaming industry often allows for more aggressive use of innovative methods. Specifically, this requires extensive research into innovative solutions (in academia and industry), and demands high-quality experimental platforms, such as rapid environment setup and comparative testing. Innovative methods also face risks in implementation due to performance and scenario limitations. For example, style transfer may be difficult to implement cost-effectively on the game client while being resource-intensive on the server side.

[0048] Fifth, there is a prevalence of solo operations, typically with one person working on a single game. Algorithm engineers often have their own methods and techniques for implementing models. For example, engineers in computer vision and natural language processing may use different engineering frameworks during project implementation. Specifically, this leads to reinventing the wheel (referring to repeatedly building the same components), independent work, a lack of code standardization, low reusability in engineering projects, and difficulty in transferring accumulated technical expertise from one framework to another, easily resulting in a situation of being general but not specialized.

[0049] 6. The models developed by algorithm engineers may rely on multiple platforms during the development and deployment process. This often requires writing glue code to connect the functions of different platforms and provide services to the game. The entire development process lacks an automated pipeline. Engineers use different frameworks for data science research and development, which may need to run in different environments. The environments and resources required for development, testing, and deployment are not the same, and each project has its own development specifications and processes. Specifically, the chain is complex: involving multiple platforms and modules, it is difficult to quickly troubleshoot which link is causing the problem. Testing is difficult: also due to the complexity of platforms and modules, test engineers usually can only perform functional and performance tests from the final service, which may mask problems. There is a lot of repetitive work: game services iterate quickly, and after each model iteration by algorithm engineers, the development process needs to be restarted, resulting in a lot of repetitive work and a high risk of problems.

[0050] It should be noted that the problems listed above are not limited to the gaming scenario; they are prevalent in various processes of applying data science to business and products. This analysis is based on the pain points faced by the gaming industry in conducting data science research and development, and extends to many common problems faced by the entire industry.

[0051] In response to the aforementioned technical problems, the inventors have discovered that if data development-related systems and model training-related systems can be built on a unified storage system, changes in the training data (including online data) during model training can be detected in real time. The data development-related systems can then process these changes to obtain data features suitable for model training, unifying the previously fragmented data development and model training processes. This eliminates the need for business-side algorithm researchers to learn multiple frameworks or concern themselves with the underlying implementation details. The frameworks can easily access storage and computing resources, bridging the boundaries between data and AI and overcoming the aforementioned problems.

[0052] To address the problems existing in the prior art, the detailed implementation of the technical solution provided in this application is as follows.

[0053] Embodiment 1 of this application provides a model training method that can be applied to a model training cluster. The model training cluster may include a data generation device and a model training device. The data generation device generates data for model training, and the model training device acquires data from the data generation device and uses the acquired data to store a training set for training the model. Specifically, the model training method provided in Embodiment 1 can be applied to the model training device within a model training cluster. In this embodiment, since the model training device and the data generation device belong to the same service cluster (i.e., the model training cluster), resource sharing can be easily achieved between them. This allows the model training device to easily acquire data from the data generation device in real time, facilitating model training based on the real-time acquired data and improving the reliability and real-time performance of the trained model.

[0054] Figure 1 This is a flowchart illustrating an embodiment of the model training method provided in this application. Figure 1 As shown, the model training method may include the following steps 11 to 14.

[0055] Step 11: Obtain the target offline data and target real-time data for training the target model from the stored training set.

[0056] The training set stores offline and real-time data used for model training. The offline data was generated more than a preset time away from the current time, while the real-time data was generated less than or equal to a preset time away from the current time. The real-time data in the training set is obtained by acquiring data generated on the data generation device in real time.

[0057] In this step, the model training device can respond to a training request for the target model by retrieving the target offline data and target real-time data from the stored training set used to train the target model. When a user has a business need, the user sends a request to the model training device through a client, indicating that the business is the training of the target model. This request can be a training request for the target model. At this time, based on the training request for the target model, the model training device retrieves the target offline data and target real-time data for training the target model from the stored training set containing relevant data for training the target model.

[0058] Optionally, the training set stores at least one offline and real-time training data for a model, both of which can be obtained through a data generating device.

[0059] For example, the time elapsed since the offline data was generated is greater than a preset time.

[0060] In one possible implementation, if the current time is 12:00 on October 15th and the preset duration is 7 days, then the offline data will be generated before 12:00 on October 8th.

[0061] For example, the time from which real-time data is generated is less than or equal to a preset time.

[0062] In one possible implementation, the current time is 12:00 on October 15th, and the preset duration is 7 days. Then the real-time data is generated between 12:00 on October 8th and 12:00 on October 15th, and includes the data generated at 12:00 on October 8th and 12:00 on October 15th.

[0063] Optionally, the target model can be a business-related model that meets the user's business needs. This business can be computer vision (CV), natural language processing (NLP), image processing, reinforcement learning, etc.

[0064] Optionally, the real-time and offline data in the training set can be obtained from various different data sources, as detailed in the following embodiments.

[0065] Optionally, the training request for the target model can be implemented by linking the client and the model training device using a Software Development Kit (SDK). In this embodiment, the model training device can also be understood as a server providing model training services.

[0066] In this embodiment, the model training device can acquire data generated by the data generating device in real time, and use the acquired data to store the training set for training the model.

[0067] Step 12: Train the model to be trained based on the target offline data and the target real-time data to obtain the trained model.

[0068] In this step, after obtaining the target offline data and target real-time data from the dataset, the model to be trained is trained and tested based on the target offline data and target real-time data to obtain the trained model.

[0069] Optionally, the training process can be as follows: First, use computing resources to process the target offline data and target real-time data to obtain online data features and offline data features. Then, based on the online data features and offline data features, train the model to be trained to obtain the trained model.

[0070] Specifically, during the training process of the model to be trained based on online and offline data characteristics, training datasets and test datasets can be constructed based on these online and offline data characteristics. Furthermore, based on container scheduling and training environment (based on Kubernetes and Docker technologies), model development, debugging, and deployment of training tasks can be carried out in a container environment. This part is detailed in the following examples.

[0071] Step 13: During the training process of the model to be trained, if it is detected that there is updated first real-time data in the training set that is used to train the target model, the first real-time data is determined as the target real-time data.

[0072] In this step, during the training of the model to be trained in step 12 above, since the training set contains data generated by the real-time data generation device, when the generated data is the updated first real-time data used to train the target model, in order to ensure the accuracy of the target model in real time, the first real-time data needs to be determined as the new target real-time data so that the trained model can be updated and trained in subsequent steps.

[0073] Optionally, when the target real-time data in the above steps are A1, A2 and A3, the training set continuously acquires data A4 generated by the data generation device. When A4 is the updated first real-time data used to train the target model, A4 is used as the target real-time data.

[0074] Step 14: In response to the detection of updated second real-time data for training the target model in the training set, update the trained model based on the second real-time data to obtain the updated trained model.

[0075] In this embodiment, the first real-time data is the data generated during the training of the model to be trained, i.e., the first real-time data is the data generated before the trained model is obtained, and the second real-time data is the data generated after the trained model is obtained. The first real-time data is generated earlier than the second real-time data. The first real-time data is used to train the model to be trained, and the second real-time data is used to update the trained model. By using the first real-time data generated during the training process to train the model to be trained, and using the second real-time data generated after training to update the trained model, the data processing accuracy of updating the trained model can be higher, the real-time performance can be better, and the reliability can be improved.

[0076] In this step, when updated second real-time data for training the target model is detected in the training set, the trained model needs to be updated using the second real-time data in order to make the generation of the target model more accurate and real-time, thus obtaining the updated trained model.

[0077] Specifically, if updated second real-time data for training the target model is detected in the training set, the second real-time data is processed using computing resources to obtain updated online data features. The updated online data features are then used to update and train the trained model to obtain the updated trained model.

[0078] The model training method provided in this application, in response to a training request for a target model, retrieves target offline data and target real-time data for training the target model from a stored training set. The training set stores offline and real-time data for model training. The generation time of the offline data is greater than a preset time interval, while the generation time of the real-time data is less than or equal to a preset time interval. The real-time data in the training set is obtained by acquiring data generated on a data generation device in real time. The model to be trained is then trained based on the target offline and target real-time data to obtain a trained model. During the training of the model to be trained, if updated first real-time data for training the target model is detected in the training set, this first real-time data is identified as the target real-time data. Finally, in response to the detection of updated second real-time data for training the target model in the training set, the trained model is updated and trained based on the second real-time data to obtain an updated trained model. This technical solution continuously updates and trains the trained model upon detecting updates to real-time data, thereby enhancing the real-time performance and reliability of model training.

[0079] In one embodiment, the model training method provided in this application can be applied to a model training cluster, which may include multiple node devices, including resource scheduling devices.

[0080] The aforementioned resource scheduling device can be any one of multiple node devices; it can also be any one of the multiple node devices that meets preset conditions, such as: available memory greater than a preset memory size, available storage space greater than a preset storage size, CPU utilization rate less than a preset utilization ratio, etc., and may also include other conditions; the resource scheduling device can also be a device selected by the operator. There can be one or multiple resource scheduling devices.

[0081] Prior to step 11, the modeling method may further include step 10a.

[0082] Step 10a: The resource scheduling device determines the first computer resources required for training the target model, and selects at least one training device for training the target model from multiple node devices based on the first computer resources and the resource usage of multiple node devices, and determines the first resource occupancy information of the training device for training the target model, so that each training device uses resources consistent with the first resource occupancy information to perform training of each model.

[0083] When there are multiple resource scheduling devices, step 10a can be executed by one resource scheduling device or by multiple resource scheduling devices working together.

[0084] The first computer resource may include at least one of the following: memory, storage space, CPU frequency and cache required for training the target model, and may also include other computer resources, which are not specifically limited in this application.

[0085] In this embodiment, the resource scheduling device can determine the first set of computer resources required for training the target model based on the computer resources required for training historically trained models. For example, if the computer resources required for training a historically trained model are: 10GB of memory, 50GB of storage space, and a CPU clock speed greater than 2.0GHz, then the first set of computing resources can be 11GB of memory, 55GB of storage space, and a CPU clock speed greater than 2.0GHz. In other words, the first set of computer resources required for training the target model can be slightly greater than the computer resources required for training historically trained models.

[0086] Alternatively, the resource scheduling device can also determine the required resources input by the operator as the first computer resources needed for training the target model. Before model training, the operator can estimate the resources required for model training and input their estimated resources into the model training cluster. In this way, the resource scheduling device can determine the required resources input by the operator as the aforementioned first computing resources. This application does not limit the specific method of determining the first computer resources.

[0087] In step 10a, the resource scheduling device can specifically select a training device from the plurality of node devices that meets the first computing resource requirements based on the first computer resource. A training device that meets the first computing resource requirements can be defined as having available computing resources that satisfy the first computing resource requirement, or having computing resource performance that satisfies the first computing resource requirement. For example, if the first computing resource requirements are: 11GB of memory, 55GB of storage space, and a CPU clock speed greater than 2.0GHz, then the selected training device that meets the first computing resource requirements will have available memory greater than 11GB, available storage space greater than 55GB, and a CPU clock speed greater than 2.0GHz.

[0088] In this step, the resource scheduling device can determine the resource information in the training device that satisfies the first computing resource requirement as the first resource occupancy information for training the target model. To ensure successful model training, the first resource occupancy information can be slightly larger than the first computing resource requirement. For example, taking the first computing resource requirement in the previous paragraph as an example, the first resource occupancy information could be 11GB of memory and 55GB of storage space.

[0089] The training devices identified in step 10a may include one, two, or more. In this embodiment, when there are multiple training devices, the training devices that meet the first computing resource requirement can be understood as the multiple training devices as a whole meeting the first computing resource requirement. For example, the available memory of the multiple training devices as a whole is greater than 11GB and the available storage space is greater than 55GB. When there are multiple training devices, the first resource occupancy information used for training the target model in each training device can be determined. The sum of the first resource occupancy information corresponding to each training device should meet the first computing resource requirement. For example, if the first computing resource is 11GB of memory and 55GB of storage space, and there are two training devices, then the first resource occupancy information of the first training device is determined to be 3GB of memory and 20GB of storage space, and the first resource occupancy information of the second training device is determined to be 9GB of memory and 40GB of storage space. The sum of the first resource occupancy information of the first training device and the first resource occupancy information of the second training device is 12GB of memory and 60GB of storage space, which meets the first computing resource requirement.

[0090] Steps 11 through 14 can be executed by each training device using resources consistent with its corresponding first resource occupancy information, as determined in step 10a. For example, the first training device executes steps 11 through 14 using 3GB of memory and 20GB of storage space, and the second training device executes steps 11 through 14 using 12GB of memory and 60GB of storage space.

[0091] This implementation pools all computer resources of each node device in the model training cluster, thereby abstracting all node devices in the model training cluster into a large server. The resource scheduling device selects the node device that meets the training requirements of the target model based on the resource usage of each node device, enabling flexible scheduling of cluster resources and making full use of the resources of each node device, thus making the model training more efficient.

[0092] In an optional embodiment, prior to step 11, the modeling method may further include steps 10b to 10d.

[0093] Step 10b: When the resource scheduling device detects the existence of real-time data to be stored, it determines the second computer resources required to store the real-time data to be stored, and selects at least one device for storing the real-time data to be stored from the multiple node devices as a data generating device based on the second computer resources and the resource usage of the multiple node devices, and determines the second resource occupancy information of the data generating device for storing the real-time data to be stored.

[0094] It is understandable that storing real-time data typically requires calculations such as storage classification and location determination. Therefore, storing data also requires the coordinated operation of the storage device's memory, storage space, and CPU. Thus, the aforementioned second computing resource usually includes, but is not limited to, memory, storage space, and CPU. The specific content of the second computing resource is similar to that of the first computing resource and will not be repeated here.

[0095] The process of determining the data generating device or the second resource occupancy information in step 10b is similar to that in step 10a, and will not be described again here.

[0096] The aforementioned real-time data to be stored can be user data sent in real time by various clients of the application, data obtained in real time from the clients by the application's server, or real-time data to be stored obtained through other means.

[0097] Step 10c: The data generating device uses resources consistent with the second resource occupancy information to store the data to be stored locally in real time.

[0098] In step 10c, each data generating device can use resources that are consistent with its corresponding second resource occupancy information to store the real-time data to be stored locally. The specific execution process is similar to the process in the above text where each training device uses resources that are consistent with its corresponding first resource occupancy information to execute steps 11 to 14, and will not be repeated here.

[0099] Step 10d: The training device acquires the data generated on the data generating device in real time, and uses the acquired data to store the training set for training the target model.

[0100] In this embodiment, the resource scheduling device selects a data generation device to store real-time data based on the resource usage of each node device in the model training cluster. This further enables flexible scheduling of cluster resources, making full use of the resources of each node device. Flexible scheduling allows data storage and model training to be carried out flexibly in the cluster, making model training and data storage more efficient. Data storage and model training can be carried out under unified resource scheduling, allowing the two processes to be carried out in a unified manner.

[0101] In this embodiment, the resource scheduling device may also be identified as a data generation device or a training device. The same node device in the model training cluster may be identified as both a data generation device and a training device. In this embodiment, resources are pooled, so the device resources are used as the allocation subject for process execution, which can make the resource utilization rate in the cluster higher and can also allocate available resources more promptly to flexibly process each task (including data storage tasks and model training tasks), further improving the efficiency of model training.

[0102] In an optional embodiment, prior to step 10a, the above model training method may further include step 10A.

[0103] Step 10A: The first device in the model training cluster packages the training program used to train the target model into a container image, the container image containing the training program and the runtime resources required to run the training program.

[0104] The first device mentioned above can be any device in the model training cluster, or a device specified by the operator in the model training cluster for building a container image, or a device that meets preset conditions, such as memory, storage space, etc. meeting preset sizes. The determination of the first device is not specifically limited in this application.

[0105] A container image is a packaged file that contains an application and the environment that the application depends on at runtime.

[0106] In step 10a, the resource scheduling device can determine the first computer resource according to the following steps: the resource scheduling device obtains the container image corresponding to the above training program from the first device, and determines the first computing resource required for the container image to run based on the running resources in the obtained container image.

[0107] This embodiment packages the training program into a container image, which makes it easier to schedule resources based on the information in the container image.

[0108] exist Figure 1 On this basis, Figure 2 This is a schematic diagram of the model training system provided in the embodiments of this application, such as... Figure 2 As shown, the system includes: a data access module 21, a data capability module 22, and a data training module 23.

[0109] exist Figure 2 On this basis, Figure 3 This is a flowchart illustrating Embodiment 2 of the model training method provided in this application. (Combined with...) Figure 2 and Figure 3 The training method for this model will be explained in detail:

[0110] Step 31: Determine the target data generation device for generating the target training data.

[0111] The target training data refers to the training data used to train the target model.

[0112] In this step, in order to train the target model, it is first necessary to determine the training data for training the target model. The training data is on the data production device, so we first need to determine the target data production device used to generate the target training data.

[0113] The data generating devices can be various data sources, such as relational database TiDB, object storage service (OSS), search engine (Elastic Search, ES), object storage S3, message queue Kafka, big data distributed file storage HDFS, document database MongoDB, etc.

[0114] Optionally, the training request carries a target identifier to represent the target model, and each data generating device has a corresponding identifier to represent the model for which the generated data is used for training.

[0115] Furthermore, this step can be implemented as follows:

[0116] Locate the target data generating device that matches the target identifier from among the various data generating devices.

[0117] In one possible implementation, the data generating device may have an identifier to indicate that the generated data is used to train the model. For example, if the target model is related to drawing, the identifier on the data generating device may be a drawing-related identifier such as "drawing", "paint", etc.

[0118] Step 32: Determine and store the training set for training the target model based on the data generated by the target data generating device.

[0119] In this step, after identifying the target data generation device, the training set for training the target model is determined in the target data generation device and stored.

[0120] Optionally, each data generating device has corresponding data recording information, which is used to record in real time the storage location of the data generated by the data generating device.

[0121] Furthermore, this step can be implemented as follows:

[0122] Step 1: Based on the target data record information corresponding to the target production device, determine the storage location of the training data used to train the target model on the target production device.

[0123] Step 2: Obtain and store the training set used to train the target model from the storage location.

[0124] In a specific implementation, such as Figure 2 As shown, the data access module 21 contains multiple namespaces (each namespace corresponds to a business, that is, a certain business can be a target model, that is, the namespace corresponds to the above-mentioned data generation device and is used to indicate the model to be trained by the generated data). Under different names, there is at least one dataset (i.e., data record information). Table 1 is a schematic table of namespaces, as shown in Table 1:

[0125] Table 1:

[0126]

[0127] Specifically, taking namespace 1 as an example, it contains 8 datasets: dataset 1 and dataset 2 under TiDB, dataset 1 and dataset 2 under OSS, dataset 1 and dataset 2 under ES, and dataset 1 and dataset 2 under S3. This multi-layered structure can effectively abstract data from different engines in a unified manner while also satisfying a certain degree of isolation. For example, datasets with similar business operations can be grouped into the same namespace.

[0128] Within the same data source, datasets must have unique names. A dataset represents a collection of data records. Although the types of data sources (which may also include message queues like Kafka, distributed file storage like HDFS, and document-oriented databases like MongoDB; it should be noted that each type of database has its own suitable scenarios—for example, relational databases are suitable for business scenarios with high requirements for consistency and low latency, document-oriented databases are suitable for scenarios with flexible model changes, and search engines utilize their unique inverted index data structure to support efficient term-based retrieval) differ, datasets within the same namespace may have different requirements for the target model, and this namespace is associated with training requests.

[0129] In other words, in this step, when responding to a training request for the target model, the corresponding dataset can be found in the corresponding namespace based on the target identifier of the target model. The dataset records the storage location of the data used to train the target model in the data source. Based on this location information, the target real-time data and target offline data are obtained from the corresponding data source.

[0130] In addition, the dataset can be configured to expose the SDK interface of the data access module 21 to the outside world. It usually includes at least the CREATE, DELETE, UPDATE, and SELECT interfaces. Depending on the data source, it may also provide corresponding advanced interfaces, such as full-text search interfaces.

[0131] For example, a namespace can be a collection of logically isolated datasets. A namespace contains a series of datasets, each of which is bound to a data source. The namespace provides a unified abstraction of datasets from different data sources, so users do not need to care about how the underlying data source manages the datasets. For example, for MySQL, there are usually several databases, and each database has several tables. By operating the dataset through the SDK, there is no need to involve the concepts of databases and tables.

[0132] Specifically, regarding the data source, it refers to a physical instance of a data engine, such as an instance of MySQL, TiDB, S3, or MongoDB. A data source is defined by multiple attributes, typically including at least:

[0133] 1. Instance host address and port number

[0134] 2. Username and password used to access the instance

[0135] 3. Additional information depending on the instance type, such as the default database connection.

[0136] Data sources typically support deployment in either standalone or cluster mode. For connected clients, both standalone and cluster deployments usually require minimal adjustments, only some additional parameters need to be modified. For development / testing environments, a standalone setup is generally used, while for production environments, a cluster mode is typically deployed to improve system availability.

[0137] Optionally, the functional architecture topology of the SDK is implemented as follows: the SDK caller accesses the SDK server through the SDK client. The SDK client can be implemented in multiple languages ​​(Python, Java, etc.). The SDK server includes basic functions such as metadata management, authentication, data access, and data development, as well as support for underlying resource scheduling and cluster management.

[0138] The SDK client and server communicate via Remote Procedure Call (RPC), supporting persistent connections to avoid frequent resource creation and destruction between the client and server. Communication between the SDK and the database cluster also uses RPC. Here, different client connection driver SDKs are typically used depending on the database. For example, Elasticsearch provides official Java / Python driver SDKs for convenient user access to Elasticsearch services. These driver SDKs usually have built-in connection pool management functionality. Connection pools prevent frequent connection creation and destruction, achieving connection and resource reuse and supporting high-concurrency access to a certain extent.

[0139] Furthermore, a data acceleration layer can be set up above the data source engine. Here, Alluxio is used. Alluxio is a cloud-based data orchestration technology for data analytics and artificial intelligence. It builds an efficient bridge between data-intensive applications and storage systems, moving data from the storage layer closer to the application, thus improving data access efficiency. Alluxio typically sits between data-driven frameworks (such as Presto and Spark) and various persistent storage systems (such as Amazon S3 and HDFS). Alluxio unifies data stored on different persistent storage systems, providing a unified client application programming interface (API) and global namespace for upper-layer applications. It's important to note that Alluxio typically supports distributed file systems and object storage, rather than OLTP databases (such as MySQL). These OLTP databases usually have built-in acceleration caches and are not the focus of Alluxio optimization.

[0140] Optionally, the training set can be stored in the form of a data lake.

[0141] like Figure 2 As shown, the data capability module 22 includes: a training set (in this embodiment, the storage of the training set can be in the form of a data lake to store offline data and real-time data).

[0142] A data lake is a centralized storage system used to store, process, and manage large amounts of structured, semi-structured, and unstructured data at any scale. Users can store data as is without prior structuring and it supports analysis based on the stored data. As shown in Table 2, compared to traditional data warehouses, data lakes have the following characteristics:

[0143] Table 2:

[0144]

[0145] Specifically, the data lake can utilize more data from more data sources in a short period of time, and can coordinate and analyze real-time and offline data in different ways when processing them later.

[0146] However, data lakes also face numerous challenges. For example, due to a lack of model control, they inevitably encounter complex data governance issues (also known as data swamps) and data access control problems. Currently, mainstream data lake implementations include Hudi, Iceberg, and Delta Lake. While each has its own strengths in terms of feature support, they all generally possess the characteristics of the aforementioned data lakes.

[0147] Therefore, in this embodiment of the application, the data lake includes: at least one data warehouse for a data theme, such as a data warehouse for image format, a data warehouse for text format, etc. When processing the data in the future, calculations are performed according to the corresponding computing resources of the data theme to achieve the effect of simple and efficient data processing.

[0148] In this step, the target offline data and target real-time data obtained in the above steps are stored in the training set, and the training set is stored in the data warehouse of each data theme in the data lake.

[0149] However, in the specific storage process, taking big data scenarios as an example, such as Figure 4 As shown, it is often necessary to synchronize data between different storage engines, such as synchronizing data in MySQL to the Hive data warehouse, that is, synchronizing data in the data access module 21 to the data lake in the data capability module 22. The traditional approach is to use two different technologies for full synchronization and incremental synchronization, and to periodically merge the fully synchronized data and the incrementally synchronized data, and finally provide services in the form of Hive.

[0150] This dual-technology approach requires two computing architectures. Therefore, for convenience and efficiency, such as... Figure 5 As shown, this application embodiment utilizes the Flink CDC function to synchronize full and incremental data from MySQL to downstream Hive or other data storage engines with a single click.

[0151] In existing data warehouse architectures, the offline and real-time data warehouses are deployed separately. The offline portion uses Spark / Hive, while the real-time portion uses Flink+Kudu or Flink+Kafka, as detailed below. Figure 6 and Figure 7As shown, the two data warehouses have two separate development processes, resulting in high maintenance costs, complex data links, and difficulty in unifying the standards for real-time and offline components because they are processed by two different engines.

[0152] At this point, in order to overcome this problem and in accordance with the concept of integrated batch processing in the following steps, Figure 6 and Figure 7 The architecture in the middle is simplified to Figure 8 As shown.

[0153] This architecture uses a single Flink SQL development platform, eliminating the need to maintain two sets of processing code. The data flow is clear, with the detail layer (ODS) performing calculations only once. Furthermore, since it uses a single codebase and engine, the data interpretation is naturally consistent. Additionally, for the Flink streaming inbound processing portion, the downstream Hive can be replaced with a data lake, better supporting data update and other operational semantics.

[0154] At this point, data lakes and Kafka can be unified, for example, using Apache Pulsar, a multi-tenant, high-performance end-to-end messaging middleware. Pulsar adopts a mainstream compute-storage separation architecture, with the top layer being the PulsarBroker (service layer). The Broker does not store any state data or perform any data storage, while the storage layer is based on Apache BookKeeper. This architecture has excellent scalability; more Producers and Consumers can be supported by extending the stateless Broker layer above, and the storage layer can be expanded to support more storage.

[0155] Compared to similar products such as Apache Kafka and Apache RocketMQ, Pulsar offers low-latency publishing and end-to-end messaging capabilities, supports millions of topics, and provides clients in various languages ​​such as Java, Go, and Python.

[0156] Step 33: In response to the detection of a change in the data in the target data generating device, update the data in the training set in real time based on the changed data in the target data generating device.

[0157] In this step, when a change is detected in the data generated by the target data generating device, the relevant record information in the data recording information will also change. At this time, based on the changed data in the target data generating device, the newly generated data is obtained in real time to update the data in the training set.

[0158] Furthermore, this step can be implemented as follows:

[0159] The training set data is updated in real time based on the changed records in the target data record information.

[0160] In one possible implementation, the changed record data in the target data record information is the storage location of the data used to train the target model on the target data generation device, and the data is retrieved from that storage location to update the data in the training set.

[0161] Step 34: Obtain the target offline data and target real-time data for training the target model from the stored training set used to train the target model.

[0162] In this step, a training set for training the target model is determined from the aforementioned data lake, and target offline data and target real-time data are extracted from the training set of the target model.

[0163] Step 35: Process the target offline data and target real-time data according to the preset data processing rules to obtain offline data features and online data features.

[0164] In this step, in order to facilitate the processing of offline and online data features in the training set, the concept of stream-batch processing needs to be introduced into the data capability module 22.

[0165] In existing technologies, stream-batch processing is a concept that extends from the Lambda and Kappa architectures in the field of big data processing. The data source may be user behavior logs or other data. In the data processing part, there are two links: one is the real-time link (also known as the acceleration layer), which generally uses a stream processing engine (such as Flink) to write data to the storage system in real time; the other is the offline link (also known as batch processing), which generally uses Spark / Hadoop MR (MapReduce) to write data to Hive (usually in a T+1 manner, for example, writing yesterday's data today), and finally uses query engines such as SparkSQL / Presto / Hive to perform ad-hoc online analysis queries (OLAP).

[0166] However, Lambda has many pain points, the first being the complexity of its architecture:

[0167] 1. Data developers have to maintain two sets of code (real-time streaming processing + offline batch processing);

[0168] 2. Data quality is difficult to control; for example, it is difficult to ensure that the logic of real-time processing and the logic of offline processing are completely consistent.

[0169] 3. When reading data, the issue of merging real-time and offline data must be considered;

[0170] 4. Two distributed systems need to be maintained.

[0171] To address the problems existing in stream computing and batch computing in the aforementioned technologies, the Kappa architecture evolved. The core of the Kappa architecture lies in using a technology stack to handle both real-time and offline data processing, and this technology stack is typically based on stream computing. For example, a streaming engine reads data from a data source (usually Kafka), processes the data in real time, or writes the data to downstream analytical databases or applications. This architecture supports real-time, near real-time, and batch processing simultaneously.

[0172] The differences between batch processing and stream computing are shown in Table 3:

[0173] Table 3:

[0174]

[0175] From the Lambda architecture to the Kappa architecture, the evolution has revolved around the core pain points of batch computing and stream computing, but the system still appears bloated. For example, there are many choices for the Storage Layer, and each storage engine has its own suitable scenarios, which in turn generates a lot of storage operation and maintenance overhead.

[0176] The integrated stream and batch architecture emerged as a result. Its core is to use the same framework to handle both batch processing and stream computing scenarios, thereby reducing learning costs, reducing resource consumption, reducing the complexity of distributed systems, and enabling faster and more efficient output of business value.

[0177] Specifically, taking the data source as the business database and business logs as an example, such as... Figure 9 As shown, in this architecture, business logs are sent to Apache Pulsar or synchronized from the business database to Pulsar via Flink CDC. The entire data chain revolves around Flink and Pulsar. Data is cleaned, transformed, merged, and aggregated by Flink (i.e., the process of processing data resources) and written to Pulsar in real time, providing data services to the outside world.

[0178] Based on the core concept of stream-batch integration, the data capability module 22 integrates Flink and mainstream data lake implementations, encapsulates the core functions required for data modeling, data storage, and data development, and also integrates a metadata center to manage metadata for various structured, semi-structured, and unstructured data. The underlying computing engine relies on data scheduling capabilities to manage computing tasks.

[0179] Optionally, at the application level: Flink provides computing resources for processing different data topics. Based on various computing resources, the target offline data and target real-time data in the data warehouse of at least one data topic are processed to obtain online data features and offline data features.

[0180] Specifically, data engineering is the process of processing and manipulating raw data, transforming raw data attributes into data features. Attributes are the inherent dimensions of data, while data features are important characteristics presented in the data, usually obtained through the calculation, combination, or transformation of attributes. For example, principal component analysis is the process of converting a large number of data attributes into a few features, which include online data features and offline data features.

[0181] In one possible implementation, the preset data processing rules could be rules for processing images, rules for processing text, rules for processing videos, and so on.

[0182] In other words, when the target offline data and the target real-time data are text, the corresponding offline data features and online data features can be obtained by using text processing rules; when the target offline data and the target real-time data are images, the corresponding offline data features and online data features can be obtained by using image processing rules; and when the target offline data and the target real-time data are videos, the corresponding offline data features and online data features can be obtained by using video processing rules.

[0183] When storing online and offline data features, corresponding online and offline feature libraries can be set up for storage.

[0184] Furthermore, the data capability module 22 exposes its functionality to the upper layer via SDK / API, providing SQL computation capabilities. In application scenarios, this upper-layer application includes areas such as image processing, natural language processing, user profiling, and reinforcement learning. The lower layer of the data capability module 22 uses a data access SDK to efficiently access data on different storage engines such as HDSF / S3 in the data access module 21.

[0185] Step 36: Identify at least one container (Docker) that matches the target identifier from among the pre-configured containers;

[0186] The training request includes a target identifier used to represent the target model.

[0187] In this step, there are multiple containers on the physical server used to process business. Each container has a container identifier, which is used to indicate the model used for training in the training environment configured for that container.

[0188] In this solution, container-based model training can be implemented using container orchestration technology (Kubernetes).

[0189] In a specific implementation, such as Figure 2 As shown, the data training module 23 can be a one-stop module that integrates model training, model evaluation / validation, model library, model service and offline service deployment capabilities, that is, it is implemented based on container and container orchestration technology.

[0190] Optionally, the data training module 23 includes: a collection of containers for training different models, which is essentially a group of Pods working together. The Pod group that constitutes the service is defined by a label selector. Kubernetes provides a service discovery mechanism by assigning IP addresses and domain names to services and load balances traffic to the network connection of the address of the Pod that matches the selector in a round-robin scheduling manner. The collection of containers includes: at least one Pod, which is the basic unit of K8s scheduling. K8s is a portable container orchestration and management tool designed for container services.

[0191] In other words, a Pod typically contains one or more containers, and each Pod is assigned a unique (within the cluster) Internet Protocol (IP) address (this allows applications to use the same port, thus avoiding conflicts). Pods support mounting data volumes, which can be network disks or local directories, and the data volume is mounted in one of the Pod's containers. Pods can be managed manually through the Kubernetes API or delegated to a controller.

[0192] Optionally, the data training module 23 also includes: Label selector: Kubernetes allows clients (users or internal components) to attach key-value pairs of labels to any API object in the system, such as a Pod or Node. The selector is used to match the query method of the object label.

[0193] Optionally, the data training module 23 also includes a controller, which is a looper used to manage a group of Pods, such as Pod replication, migration, scaling, etc. For example, when a host machine crashes, the controller can quickly create a replacement Pod. The controller is the core of the Kubernetes system.

[0194] In one possible implementation, upon receiving an instruction to train the target, the label controller determines the target container set corresponding to the data features from at least one set of container sets based on the data features, and assigns the same IP address to each container in the target server so that it can be called by each thread in the application during the subsequent model training process.

[0195] In addition, the data training module 23 is built based on key-value storage components, API servers, schedulers, control managers, Node nodes (Kubernetes nodes), Kube agents, and agent components for resource usage and performance metrics.

[0196] Specifically, the components will be introduced as follows:

[0197] Firstly, the key-value storage component can be Etcd, a lightweight, distributed, and strongly consistent key-value storage component developed by CoreOS for reliably storing cluster configuration data. This component can represent the overall state of the cluster at any given time, and etcd will synchronize the corresponding state when the cluster state changes.

[0198] Secondly, the API server provides an internal and external interface to Kubernetes using RestHTTP. The API server accepts and verifies Rest requests, and then updates the state stored in etcd.

[0199] Thirdly, the scheduler is a pluggable component that selects which node an unscheduled Pod should be scheduled to based on resource availability. The scheduler tracks the resource utilization on each node to ensure that the workload on each node does not exceed the available resources.

[0200] Fourthly, the control manager includes the DaemonSet process controller and the replication controller. This control manager communicates with the API server to create, update, and delete managed resources (Pods, server endpoints, etc.) as needed.

[0201] 5. Node (Kubernetes node): Also known as a Worker, it is a single machine (or virtual machine) used to deploy containers (workloads). Each node in the cluster must have a container runtime environment, such as Docker.

[0202] 6. Kube proxy is a concrete implementation of network proxy and load balancing. It abstracts services and supports most network operations. Based on the requested IP and port, Kube proxy will forward traffic to the specified appropriate container.

[0203] 7. The agent component for resource usage and performance metrics can be cAdvisor, which is used to monitor and collect information such as CPU, memory, network, and disk I / O of containers on each node.

[0204] In one possible implementation, during the training of the target model, Etcd synchronizes the corresponding state to the API server when the cluster state changes. The scheduler is used to schedule and select the containers required for specific model training based on information such as resource utilization on each node. cAdvisor monitors and collects information such as CPU, memory, network, and disk I / O of the containers on each node. During the training of the model, when new traffic occurs, the Kube agent forwards the indication corresponding to the traffic to the designated container.

[0205] In addition, the data training module 23 provides online and offline service deployment capabilities. Users can deploy servers developed based on server-side frameworks such as Spring on the data training module 23, and can also utilize the scheduling and computing capabilities of the AI ​​system to run online and offline jobs. The upper layer of the data training module 23 exposes an SDK / API for external systems to call.

[0206] Step 37: Using offline and online data features, train the model to be trained in at least one container to obtain the trained model.

[0207] In this step, after determining at least one container for training the model to be trained, the model to be trained is trained based on the offline and online data features determined in the above steps to obtain the trained model.

[0208] Furthermore, at least one container set that matches the target identifier is determined from a plurality of pre-configured container sets, wherein each container in the same container set is configured with the same training environment, and each container set has a container identifier.

[0209] That is, the step can be: training the model to be trained in at least one set of containers.

[0210] Based on the runtime environment provided by the container set, the containers in the container set work together to train the model to be trained in the corresponding container using offline data features and real-time data features, thus obtaining the trained model.

[0211] The development and deployment of machine learning / deep learning models typically involves several stages: data preparation (including data ingestion and feature engineering), model training, model evaluation, and model deployment. Depending on whether the model's performance meets the required standards and whether it can satisfy online business needs, this process may be repeated many times for continuous optimization. After training, the model is usually uploaded to a model repository, which is a tool similar to network storage that can centrally store large models and facilitate their subsequent retrieval.

[0212] Optionally, inference services within the target model building service are typically provided to external services via APIs / SDKs. For example, the model can be wrapped in a web framework and exposed as a REST HTTP API, allowing external processes to make remote method calls via Hypertext Transfer Protocol (HTTP). Model services can be categorized as online or offline. Offline services refer to manual or scheduled batch processing of inference tasks, while online services are directly used in production lines and require real-time performance. Depending on the deployment method, model services can be further divided into embedded and standalone types. In embedded mode, the inference service is directly integrated into the business system as a function or module, with the host system handling the inference service calls. In standalone mode, the model service and the business system are independent of each other, communicating via Remote Procedure Call (RPC) without intruding on the host system environment.

[0213] Furthermore, the container scheduling and training environment of the data training module 23 are built on Kubernetes and Docker technologies. Container scheduling provides the ability to manage any container on any node of the Kubernetes cluster, and the training environment is built on Docker containers. Users can build new containers by writing Dockerfiles (a configuration file used to describe the Docker environment) and develop, debug, and deploy code and training tasks in the container environment.

[0214] The data training module 23 processes data through the underlying data access SDK. Data generally comes from two sources: an online feature library, used to store real-time feature data at the user level (such as a user's spending amount over the past 7 days); and an offline feature library, used to store feature insights derived from processing large amounts of historical data (such as a user's spending habits over the past year). The data training module 23 can also directly retrieve data from the underlying physical storage engine of the data access module 21 through the data access SDK, offering high flexibility.

[0215] Furthermore, the specific steps for updating the trained model based on the second real-time data to obtain the updated trained model are as follows (for details not disclosed, please refer to the above embodiments):

[0216] Step 1: Process the second real-time data according to the preset data processing rules to obtain the updated online data features;

[0217] Step 2: Based on the updated online data features, update the trained model to obtain the updated trained model.

[0218] Furthermore, the technical solutions in the above embodiments can be based on the principles of Continuous Integration, Continuous Delivery, or Continuous Deployment (CICD). When a step changes, the CICD system automatically packages the software into a deliverable form, automated testing is automatically triggered, and the software is automatically deployed online to provide external services after the tests pass. CICD systems can improve the efficiency of software development and deployment, and identify problems exposed during the software development process as early as possible.

[0219] That is, based on this CICD, each time the data to be processed and the processing model for processing the data to be processed are obtained (the processing process of the above embodiment is executed each time the data is obtained), the processing model can be used to process the data to be processed and obtain the processing result.

[0220] CICD is the foundation of DevOps. DevOps refers to a set of best practices for software development and operations. It aims to shorten the software development cycle and continuously deliver high-quality software. DevOps is usually closely integrated with Agile Software Development, and many best practices in DevOps come from Agile Development.

[0221] MLOps, similar to DevOps, refers to a set of best practices for deploying and maintaining machine learning models, aiming to deploy these models more reliably and efficiently online. Under the definition of MLOps, machine learning models are developed and tested in an independent sandbox before being deployed online. The entire process is pipeline-based. MLOps bridges the gap between data scientists, DevOps professionals, and machine learning engineers, aiming to improve the automation level and quality of the entire process from model development and testing to deployment while meeting business needs. MLOps covers the entire lifecycle of machine learning models and has gradually evolved from a set of best practices into a systematic methodology. It's important to note that MLOps is a subset of ModelOps; MLOps refers to the process of developing machine learning models, while ModelOps refers to all AI models in a generalized sense.

[0222] The development of MLOps has gone through the following stages, as shown in Table 4:

[0223] Table 4:

[0224]

[0225]

[0226] The third phase of the MLOps architecture is as follows: Data scientists acquire data features from a data feature library, conduct exploratory analysis based on these features, and build model code (usually based on model development frameworks such as Tensorflow and PyTorch). Then, they submit the model building code to the code repository. The CI system automatically pulls the latest code, triggering the building, testing, and assembly of the modules (packages) needed for subsequent pipelines. The CD system deploys these modules onto an automated pipeline, which typically includes stages such as data extraction, data validation, data preparation, model training, model evaluation, and model validation. The pipeline accesses the central model registry and pushes the final models ready for deployment to the central repository (Trainedmodel). The ModelServing service pulls the models and deploys the inference service, which may always obtain the model's input data from the online feature library. Simultaneously, the system supports model performance monitoring. Based on monitoring performance metrics, it may trigger new training to iterate the model or return to the initial data analysis phase to redo the entire process.

[0227] also, Figure 2 The system shown connects the data capability module 22 and the data training module 23, achieving a closed-loop data system. For example, the data preparation phase utilizes the computing power of the data capability module 22; model training retrieves data from the offline and online feature libraries of the data capability system and performs training in the training environment provided by the data training module 23; and model services rely on the model library and online service capabilities of the data training module 23. With this pipeline system, the business side can quickly develop, test, deploy, and launch new machine learning / deep learning models, rapidly conduct A / B testing, and continuously iterate the model based on accumulated data and model performance monitoring.

[0228] That is, Figure 2 The system shown can be a data science automated pipeline, adopting a mainstream layered architecture. The bottom layer is the data access module 21 (storage engine and data access SDK), the middle layer is the data capability module 22 and the data training module 23, and the upper layer is based on the API exposed by the data training module 23 to connect to various business scenarios.

[0229] The data science automation pipeline also realizes a closed loop from business to platform. Business-side developers can complete the entire process from data processing, model training and optimization to service deployment based on the platform, and can monitor and analyze the model's performance based on the metrics collected by the platform and then make targeted iterations.

[0230] The model training method provided in this application embodiment determines a target data generation device for generating target training data, determines and stores a training set for training a target model based on the data generated by the target data generation device, updates the data in the training set in real time in response to detecting changes in the data in the target data generation device based on the changed data, obtains target offline data and target real-time data for training the target model from the stored training set, processes the target offline data and the target real-time data according to preset data processing rules to obtain offline data features and online data features, determines at least one container consistent with the target identifier from a pre-configured plurality of containers, and trains the model to be trained in the at least one container using the offline data features and online data features to obtain the trained model. In this technical solution, target real-time data and target offline data are obtained from the namespace related to the model to be trained to achieve real-time performance and reliability of model training.

[0231] Furthermore, Figure 10 This is a flowchart illustrating the data processing method provided in an embodiment of this application. The specific execution steps of this method are as follows:

[0232] Step 101: Obtain the data to be processed and the processing model used to process the data to be processed.

[0233] The processing model was trained according to the model training method described in the above embodiments.

[0234] Step 102: Use the processing model to process the data to be processed and obtain the processing results.

[0235] Optional, Figure 10 The embodiment shown is a practical application after obtaining the processing module implemented according to the above method.

[0236] The data processing method provided in this application acquires the data to be processed and a processing model for processing the data, then uses the processing model to process the data to obtain the processing result. This technical solution, based on the aforementioned model training method and CICD (Computer Integrated Development and Control) concept, improves the data processing capability and efficiency.

[0237] Furthermore, Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 11As shown, the electronic device may include: a processor 111, a memory 112, and computer program instructions stored in the memory 112 and executable on the processor 111.

[0238] Processor 111 executes computer execution instructions stored in memory 112, causing processor 111 to perform the scheme in the above embodiments. Processor 111 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0239] The memory 112 is connected to the processor 111 via the system bus and communicates with it. The memory 112 is used to store computer program instructions.

[0240] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0241] The electronic device provided in this application embodiment can be used to execute the technical solution corresponding to the model training method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0242] This application also provides a chip for executing instructions, which is used to execute the technical solution of the model training method in the above embodiments.

[0243] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are run on a server, the server executes the technical solution of the model training method described in the above embodiments.

[0244] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to perform the technical solution of the model training method described in the above embodiments.

[0245] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to general-purpose or special-purpose servers.

[0246] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0247] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A model training method, characterized in that, The method is applied to a model training cluster, the model training cluster includes a plurality of node devices, the plurality of node devices include a resource scheduling device, the model training cluster includes a data generation device and a model training device, and the method includes the following steps: The resource scheduling device determines second computer resources required for storing the to-be-stored real-time data when detecting that there is to-be-stored real-time data, and selects at least one device from the plurality of node devices as a data generation device for storing the to-be-stored real-time data according to the second computer resources and resource usage of the plurality of node devices, and determines second resource occupation information of the data generation device for storing the to-be-stored real-time data; The data generation device stores the to-be-stored real-time data into the local device using resources consistent with the second resource occupation information; The model training device acquires data generated by the data generation device in real time, and stores a training set for training a target model according to the acquired data in real time; The model training device acquires target offline data and target real-time data for training the target model from the stored training set for training the target model, the real-time data in the training set is obtained by acquiring data generated by the data generation device in real time, the generation time of the offline data in the training set is greater than a preset time length from the current time, and the generation time of the real-time data in the training set is less than or equal to the preset time length from the current time; The model training device trains a to-be-trained model according to the target offline data and the target real-time data, and obtains a trained model; During the process of training the to-be-trained model, if the model training device detects that there is updated first real-time data for training the target model in the training set, the model training device determines the first real-time data as the target real-time data; The model training device updates and trains the trained model based on the second real-time data to obtain an updated trained model in response to detecting that there is updated second real-time data for training the target model in the training set.

2. The method of claim 1, wherein, Before the step of acquiring target offline data and target real-time data for training the target model from the stored training set for training the target model, the method further includes the following steps: The resource scheduling device determines first computer resources required for training the target model, and selects at least one training device for training the target model from the plurality of node devices according to the first computer resources and resource usage of the plurality of node devices, and determines first resource occupation information of the training device for training the target model, so that each training device uses resources consistent with the first resource occupation information to train each model.

3. The method of claim 2, wherein, Before the resource scheduling device determines the first computer resources required for training the target model, the method further includes the following steps: A first device in the model training cluster packs a training program for training the target model into a container image, and the container image contains the training program and running resources required for running the training program; The resource scheduling device determines first computer resources required for target model training, including: The resource scheduling device determines first computing resources required by the container image runtime according to the running resources.

4. The method of claim 1, wherein, Before the target offline data and the target real-time data for training the target model are obtained from the stored training set for training the target model, the method further includes: Determining a target data generation device for generating target training data, the target training data being training data for training the target model; Determining and storing a training set for training the target model according to data generated by the target data generation device; In response to detecting a change in data in the target data generation device, updating data in the training set in real time according to the changed data in the target data generation device.

5. The method of claim 4, wherein, The target offline data and the target real-time data for training the target model are obtained from the stored training set for training the target model, including: In response to a training request for a target model, obtaining target offline data and target real-time data for training the target model from the stored training set for training the target model; The training request carries a target identifier for representing the target model; each data generation device corresponds to an identifier for representing the model for which the generated data is used for training; The determination of the target data generation device for generating target training data includes: Finding a target data generation device consistent with the target identifier from each data generation device.

6. The method of claim 4, wherein, Each of the data generation devices corresponds to data record information, which is used to record the storage location of the data generated by the data generation device in real time; The determination and storage of the training set for training the target model according to the data generated by the target data generation device includes: According to the target data record information corresponding to the target data generation device, determining the storage location of the training data for training the target model on the target data generation device; Obtaining and storing the training set for training the target model from the storage location; The real-time updating of the data in the training set according to the changed data in the target data generation device includes: According to the record data changed in the target data record information, updating the data in the training set in real time.

7. The method according to claim 1 or 2, characterized in that, The training of the to-be-trained model according to the target offline data and the target real-time data to obtain a trained model includes: Processing the target offline data and the target real-time data according to a preset data processing rule to obtain offline data features and online data features; Training the to-be-trained model according to the offline data features and the online data features to obtain a trained model; The update training of the trained model based on the second real-time data to obtain an updated trained model includes: Processing the second real-time data according to the preset data processing rule to obtain updated online data features; According to the updated online data features, the post-training model is updated and trained to obtain an updated post-training model.

8. The method of claim 7, wherein, The target offline data and the target real-time data used for training the target model are obtained from a stored training set used for training the target model, and the method comprises the following steps of: In response to a training request for a target model, target offline data and target real-time data used for training the target model are obtained from a stored training set used for training the target model, and the training request carries a target identifier used to represent the target model; According to the offline data features and the online data features, the to-be-trained model is trained to obtain a post-training model, and the method comprises the following steps of: At least one container consistent with the target identifier is determined from a plurality of pre-configured containers, and each container has a container identifier used to represent a model trained in a training environment configured by the container; The to-be-trained model is trained in the at least one container using the offline data features and the online data features to obtain a post-training model.

9. The method of claim 8, wherein, At least one container consistent with the target identifier is determined from a plurality of pre-configured containers, and each container has a container identifier used to represent a model trained in a training environment configured by the container; At least one container set consistent with the target identifier is determined from a plurality of pre-configured container sets, and each container in the same container set has a same training environment, and each container set has a container identifier; The to-be-trained model is trained in the at least one container set. The method comprises:

10. A data processing method, characterized by, Obtaining to-be-processed data and a processing model used for processing the to-be-processed data, wherein the processing model is obtained by training according to the model training method in any one of claims 1 to 9; Processing the to-be-processed data using the processing model to obtain a processing result. The system is applied to a model training cluster, the model training cluster comprises a plurality of node devices, the plurality of node devices comprise a resource scheduling device, the model training cluster comprises a data generation device and a model training device, and the system comprises a data capability module and a data training module.

11. A model training system, comprising: ​ The data capability module is configured to acquire target offline data and target real-time data for training a target model from a stored training set for training the target model by a model training device, real-time data in the training set being obtained from data generated on a real-time data acquisition device, offline data in the training set being generated at a time distance from a current time greater than a preset time length, and real-time data in the training set being generated at a time distance from the current time less than or equal to the preset time length; wherein, when the resource scheduling device detects that there is to-be-stored real-time data, the resource scheduling device determines second computer resources required for storing the to-be-stored real-time data, selects at least one device from the plurality of node devices as a data generation device for storing the to-be-stored real-time data according to the second computer resources and resource usage of the plurality of node devices, and determines second resource occupation information of the data generation device for storing the to-be-stored real-time data; the data generation device stores the to-be-stored real-time data in the local using resources consistent with the second resource occupation information; and the model training device acquires data generated on the data generation device in real time and stores a training set for training the target model according to the acquired data in real time; The data training module is configured to train a to-be-trained model according to the target offline data and the target real-time data by the model training device to obtain a trained model; The data capability module is further configured to, by the model training device, if it is detected that there is first real-time data that is updated and used for training the target model in the training set during the training of the to-be-trained model, determine the first real-time data as the target real-time data; The data training module is further configured to, by the model training device, in response to detecting that there is second real-time data that is updated and used for training the target model in the training set, perform update training on the trained model based on the second real-time data to obtain an updated trained model.

12. An electronic device, comprising: Comprise: a processor, a memory, and computer program instructions stored on the memory and executable on the processor; The processor executes the computer program instructions to implement the model training method of any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the model training method of any one of claims 1-10.

Images