Data processing method and device based on TLMP platform

By using message queues and multiple Log write threads for asynchronous Log information storage on the TLMP platform, the problem of low model training speed caused by Log information storage is solved, and an efficient model training process is realized.

CN119026701APending Publication Date: 2024-11-26MOMENTA (SUZHOU) TECHNOLOGY CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202310581229.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

During the machine learning model training process based on the TLMP platform, the storage of Log information causes the model training speed to be low, which takes a long time to complete the storage and continue to the next step, affecting the overall training efficiency.

Method used

Use message queues to store asynchronous Log information. After iteration training is completed, the Log information is stored in the message queue in timestamp order, and it is efficiently stored in the database through multiple Log write thread polling and weighting strategies, avoiding waiting delays and improving training efficiency .

Benefits of technology

By storing Log information asynchronously, the efficiency of model training is significantly improved, the Log write thread crashes are avoided, and the model training speed is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026701A_ABST
    Figure CN119026701A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of machine learning, and discloses a data processing method and device based on a TLMP platform. The method comprises the steps of sequentially executing multiple iterative training of a machine learning model in a wait-free manner based on a TLMP platform until model training is completed; wherein a plurality of pieces of Log information generated by each time of iterative training are stored according to the following mode: after each time of iterative training is finished, calling a corresponding Log interface to sequentially store the plurality of pieces of Log information generated by the current time of iterative training into a message queue according to a timestamp, and then sending a write-in instruction to a Log write-in thread so as to store each piece of Log information in the message queue in a database. According to the method, in machine learning model development based on a TLMP platform, a large amount of Log information storage does not need to be waited before next iteration training; and the model training speed is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a data processing method and device based on a TLMP platform. Background Art

[0002] The TLMP (Training Lifecycle Management Platform) platform is used to achieve standardized development of machine learning models.

[0003] When developing machine learning models based on various TLMP platforms, it generally includes the data preparation stage, the model training stage, and the final deployment stage. In each stage, in order to facilitate the query of relevant code, configuration, results and other data and the clarification of anomalies in model development, the log information (log information) corresponding to each step in each stage is generally stored. In the existing TLMP platform, the writing and storage of log information generally adopts the following process: after the corresponding step is completed, the generated data is uniformly stored, and the next step can be continued after the storage is completed.

[0004] However, in the model training phase, a large amount of data is generated after each step of training, and the unified storage of the log information corresponding to the large amount of data and the training itself will consume a lot of time, that is, it takes a long time before the next step of training can be continued. This increases the training cycle of the entire model when storing log information, which makes it difficult to improve the training speed of the model in this case. Summary of the invention

[0005] The purpose of this application is to provide a data processing method and device based on a TLMP platform to solve the technical problem of low model training speed caused by Log storage during current machine learning model training.

[0006] To achieve the above objectives, this application proposes the following technical solutions:

[0007] In a first aspect, the technical solution provides a data processing method based on a TLMP platform, comprising:

[0008] Based on the TLMP platform, multiple iterations of machine learning model training are executed sequentially without waiting until the model training is completed;

[0009] Among them, the several log information generated by each iterative training is stored in the following manner: after each iterative training is completed, the corresponding Log interface is called to store the several log information generated by this iterative training in the message queue in sequence according to the timestamp, and then a write instruction is sent to the Log writing thread to store each log information in the message queue in the database.

[0010] Furthermore, after calling the corresponding Log interface to sequentially store the plurality of Log information generated by the iterative training into the message queue according to the timestamp, the method further includes:

[0011] The log writing thread polls the message queue to sequentially obtain each log information from the message queue and stores it in the database.

[0012] Furthermore, storing each of the Log information in the message queue in a database includes:

[0013] Obtain the total number of logs of the plurality of log information in the message queue;

[0014] When it is determined that the total number of logs is greater than the sum of the maximum processing amounts of all log writing threads, each log writing thread acquires log information several times according to its own maximum processing amount and stores it in the relational database until the remaining log information is less than the sum of the maximum processing amounts of all log writing threads, and the remaining log information is acquired and stored in the relational database using the ratio of the maximum processing amount of the log writing thread to the sum of the maximum processing amounts of all log writing threads as a weight;

[0015] Otherwise, the Log information is obtained from the message queue and stored in the relational database using the ratio of the maximum processing volume of the Log writing thread to the sum of the maximum processing volumes of all Log writing threads as a weight.

[0016] Furthermore, after the plurality of log information generated by each iterative training is stored in the following manner, it includes:

[0017] Evaluating the current iterative training of the machine learning model;

[0018] The evaluation results are compared with the expected results to obtain the parameter quantity whose deviation between the operating value and the expected value is greater than the deviation threshold, and based on the parameter quantity, the corresponding Log information is retrieved from the database to the visualization front end.

[0019] Furthermore, storing each of the Log information in the message queue in a database includes:

[0020] When it is determined based on the unique field in the database that the Log information to be stored is repeated with the existing Log information in the database, discarding the Log information to be stored;

[0021] Otherwise, the Log information to be stored is inserted into the database.

[0022] In a second aspect, the technical solution provides a data processing device based on the TLMP platform, comprising:

[0023] An execution module, used to sequentially execute multiple iterations of the machine learning model training without waiting based on the TLMP platform until the model training is completed;

[0024] The execution module includes the following units to implement the storage of several Log information generated by each iterative training:

[0025] A first Log storage unit is used to call the corresponding Log interface after each iteration training to store the plurality of Log information generated by the iteration training into the message queue in sequence according to the timestamp;

[0026] The second Log storage unit is used to send a write instruction to the Log writing thread to store each of the Log information in the message queue in the database.

[0027] Furthermore, the execution module also includes:

[0028] The third Log storage unit is used for the Log writing thread to poll the message queue to sequentially obtain each Log information from the message queue and store it in the database.

[0029] Furthermore, the execution module includes:

[0030] A statistics acquisition unit, used to acquire the total number of logs of all log information in the message queue;

[0031] The first judgment unit is used to judge that when the total number of logs is greater than the sum of the maximum processing amounts of all log writing threads, each log writing thread obtains log information several times according to its own maximum processing amount and stores it in the relational database until the remaining log information is less than the sum of the maximum processing amounts of all log writing threads, and then obtains the remaining log information and stores it in the relational database using the ratio of the maximum processing amount of the log writing thread to the sum of the maximum processing amounts of all log writing threads as a weight;

[0032] Otherwise, the Log information is obtained from the message queue and stored in the relational database using the ratio of the maximum processing volume of the Log writing thread to the sum of the maximum processing volumes of all Log writing threads as a weight.

[0033] In a third aspect, the technical solution provides an electronic device, comprising at least one processor, wherein the processor is coupled to a memory, wherein a computer program is stored in the memory, and wherein the computer program is configured to execute the method described when executed by the processor.

[0034] In a fourth aspect, the present technical solution provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to execute the described method.

[0035] It can be seen from the above technical solutions that the technical solution of the present application provides a data processing method based on the TLMP platform to improve the technical defect of the existing TLMP platform that the model training speed is reduced due to the storage of Log information when performing data processing.

[0036] The technical scheme includes: first, completing an iterative training of a machine learning model based on the TLMP platform, and obtaining several Log information corresponding to the iterative training. Secondly, executing the next iterative training in a loop, and calling the corresponding Log interface to store each of the Log information in the message queue in sequence according to the timestamp. Then, sending a write instruction to the Log writing thread so that it can sequentially obtain each of the Log information from the message queue and store it in the database. It can be seen that in order to avoid the time delay caused by the Log information storage to enter the next iterative training after the end of an iterative training, the message queue is introduced when the Log information storage corresponding to this iteration is performed. At this time, it is only necessary to store the Log information in the message queue and then return to continue to perform the next iterative training. Then, when performing the next iterative training, the acquisition and storage of the Log information in the message queue are realized based on the Log writing thread. Then, through the asynchronous execution of iterative training and Log information storage, the time waste caused by directly storing the Log information in the database through the Log writing thread after the end of this iterative training in the prior art is avoided. The efficiency of model training is greatly improved. At the same time, in order to avoid the crash of the Log writing thread due to too much Log information, a method of actively obtaining Log information from the message queue based on the write instruction is adopted.

[0037] It should be appreciated that all combinations of the foregoing concepts, as well as additional concepts described in greater detail below, may be considered to be part of the inventive subject matter of the present disclosure, provided such concepts are not mutually inconsistent.

[0038] The foregoing and other aspects, embodiments and features of the present application can be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the present application, such as the features and / or beneficial effects of the exemplary embodiments, will be apparent from the following description or learned through practice of the specific embodiments according to the teachings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures may be represented by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the present application will now be described by way of example and with reference to the accompanying drawings, in which:

[0040] Figure 1 is a flow chart of the data processing method based on the TLMP platform described in this embodiment;

[0041] Figure 2 This is another flowchart for storing log information;

[0042] Figure 3 This is a flowchart for log storage based on multiple consumers;

[0043] Figure 4 Flowchart to prevent duplicate consumption of Log information;

[0044] Figure 5 A flowchart showing the association between model training results and Log information;

[0045] Figure 6 It is a block diagram of the data processing method based on the TLMP platform described in this embodiment;

[0046] Figure 7 Schematic diagram of the structure of the electronic device described in this embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the embodiments of the present application will be clearly and completely described below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meanings understood by people with general skills in the field to which this application belongs.

[0048] The words "first", "second" and similar words used in the patent application specification and claims of this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular form of "a", "an" or "the" and other similar words do not indicate a quantitative limitation, but rather indicate the presence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before "include" or "comprise" include the features, wholes, steps, operations, elements and / or components listed after "include" or "comprise", and do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0049] Although various TLMP platforms have been developed in the prior art to achieve systematic and standardized development of machine learning models, the large amount of log information stored after each iteration of training often reduces the model training speed, which in turn has an adverse effect on model development. Based on this, this embodiment aims to provide a data processing method based on the TLMP platform to improve the adverse effect of the above-mentioned log information storage on model training efficiency.

[0050] The following is a detailed introduction to the data processing method based on the TLMP platform disclosed in this embodiment in conjunction with the accompanying drawings. The method includes:

[0051] Step S102: Execute multiple iterations of training of the machine learning model sequentially without waiting based on the TLMP platform until the model training is completed;

[0052] The log information generated by each iteration of the training is stored as follows:

[0053] First, after each iteration of the training is completed, the corresponding Log interface is called to store the plurality of Log information generated by the iteration of the training into the message queue in sequence according to the timestamp.

[0054] Secondly, a write instruction is sent to the Log write thread to store each of the Log information in the message queue in the database.

[0055] The "no waiting" mentioned in this step means that after one iteration training is completed, the next iteration training can be directly entered without waiting for the Log information generated by the current iteration training to be stored in the database.

[0056] In this embodiment, the producer of the message queue is the Log function in the model training. When the TLMP platform is the MLflow platform, the Log interface for recording the parameter quantity corresponding to a single key and value is mlflow.log_param(), the Log interface for recording the matrix corresponding to a single key and value is mlflow.log_metric(), and the Log interface for recording the corresponding to a local file or path is mlflow.log_artifact().

[0057] When the log information of this embodiment is stored in the database, the producer of the message queue actively informs the consumer to consume the log information, that is, actively obtains the log information from the message queue based on the write instruction. This avoids the crash of the log writing thread caused by too much log information. In this embodiment, the write instruction is specifically a Logging signal.

[0058] As an optional implementation, Figure 2 As shown, the consumer can also actively poll to consume the Log information in the message queue.

[0059] Step S10202: The log writing thread polls the message queue to obtain each log information from the message queue in sequence and stores it in the database.

[0060] As a preferred implementation, in order to improve the consumption efficiency of the log information in the message queue, this embodiment includes several log writing threads. At this time, in order to improve the logic of each log writing thread when acquiring and storing log information, such as Figure 3 As shown, the following steps are also included:

[0061] Step S10222: Obtain the total number of logs of all log information in the message queue.

[0062] Step S10224: when it is determined that the total number of logs is greater than the sum of the maximum processing amounts of all log writing threads, each log writing thread acquires log information several times according to its own maximum processing amount and stores it in the relational database until the remaining log information is less than the sum of the maximum processing amounts of all log writing threads, and the remaining log information is acquired and stored in the relational database using the ratio of the maximum processing amount of the log writing thread to the sum of the maximum processing amounts of all log writing threads as a weight;

[0063] Otherwise, the Log information is obtained from the message queue and stored in the relational database using the ratio of the maximum processing volume of the Log writing thread to the sum of the maximum processing volumes of all Log writing threads as a weight.

[0064] In steps S10222 to S10224, effective acquisition of the Log information in the message queue is achieved based on the maximum processing volume of each Log writing thread and the weight of the Log writing thread determined by the maximum processing volume.

[0065] As another preferred implementation, in order to avoid repeated consumption of Log information in the message queue, Figure 4 As shown, it also includes:

[0066] Step S10242: When it is determined based on the unique field in the database that the Log information to be stored is repeated with the existing Log information in the database, the Log information to be stored is discarded; otherwise, the Log information to be stored is inserted into the database.

[0067] As a further implementation method, the Log information is directly associated with the iterative training process to improve the model training efficiency. Figure 5 As shown, the following steps are also included:

[0068] Step S10262: Evaluate the current iterative training of the machine learning model.

[0069] Step S10264, compare the evaluation results with the expected results to obtain the parameter quantity whose offset between the operating value and the expected value is greater than the offset threshold, and based on the parameter quantity, retrieve the corresponding Log information from the database to the visualization front end.

[0070] At this time, the log information corresponding to the abnormal parameter quantity can be automatically retrieved and visualized to avoid the problem of low efficiency in abnormal clarification caused by manual log search, which improves the efficiency of model training from another perspective.

[0071] It can be seen from the above implementation process that the data processing method based on the TLMP platform described in this embodiment realizes the asynchronous storage of the corresponding Log information after one iteration training is completed compared with the next iteration training through the message queue, thereby avoiding the time consumption of the training process and improving the training efficiency. Considering the problem of repeated consumption of Log information in the training queue, the constraint conditions in the database are introduced. Considering the consumption efficiency of consumers, multiple consumers are created; and considering the logic of the multiple consumers when consuming Log information, a method for obtaining Log information based on the maximum processing volume of consumers is designed. In addition, considering the correlation between Log information and model performance, the automatic acquisition and visual display of Log information of the parameter quantity corresponding to the running offset is realized, thereby improving the efficiency of model training from another perspective. Therefore, the training speed and training effect of the model training based on the method described in this embodiment are further improved overall.

[0072] The above program can be run in the processor, or it can be stored in the memory (or computer-readable storage medium), which includes permanent and non-permanent, removable and non-removable media. Information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable storage media does not include temporary computer-readable media, such as modulated data signals and carrier waves.

[0073] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0074] Based on this, this embodiment also provides a data processing device based on the TLMP platform. Figure 6 As shown, the device comprises:

[0075] An execution module, used to sequentially execute multiple iterations of the machine learning model training without waiting based on the TLMP platform until the model training is completed;

[0076] Among them, the several log information generated by each iterative training is stored in the following manner: after each iterative training is completed, the corresponding Log interface is called to store the several log information generated by this iterative training in the message queue in sequence according to the timestamp, and then a write instruction is sent to the Log writing thread to store each log information in the message queue in the database.

[0077] The device is used to implement the steps of the above method, so what has been described has not been repeated here.

[0078] For example, the execution module includes:

[0079] The first Log storage unit is used for the Log writing thread to poll the message queue to sequentially obtain each Log information from the message queue and store it in the database.

[0080] For another example, the execution module includes:

[0081] The acquisition unit is used to acquire the total number of logs of all log information in the message queue.

[0082] The first judgment unit is used to judge that when the total number of logs is greater than the sum of the maximum processing amounts of all log writing threads, each log writing thread obtains log information several times according to its own maximum processing amount and stores it in the relational database until the remaining log information is less than the sum of the maximum processing amounts of all log writing threads, and then obtains the remaining log information and stores it in the relational database using the ratio of the maximum processing amount of the log writing thread to the sum of the maximum processing amounts of all log writing threads as a weight;

[0083] Otherwise, the Log information is obtained from the message queue and stored in the relational database using the ratio of the maximum processing volume of the Log writing thread to the sum of the maximum processing volumes of all Log writing threads as a weight.

[0084] For example, the device comprises:

[0085] An evaluation module, used to evaluate the current iterative training of the machine learning module;

[0086] The judgment module is used to compare the evaluation results with the expected results to obtain the parameter quantity whose offset between the operating value and the expected value is greater than the offset threshold, and based on the parameter quantity, retrieve the corresponding Log information from the database to the visualization front end.

[0087] For another example, the execution module further includes:

[0088] The second judgment unit is used to discard the Log information to be stored if it is determined based on the unique field in the database that the Log information to be stored is repeated with the existing Log information in the database; otherwise, the Log information to be stored is inserted into the database.

[0089] Since the device is built based on the method, during actual operation, it can also effectively avoid the low efficiency of overall model training caused by the storage of corresponding log information after one training is completed, that is, it is guaranteed that based on the device, a higher model training speed can always be achieved when model training is performed through the TLMP platform.

[0090] This embodiment also provides an electronic device. Figure 7 As shown, it includes at least one processor, the processor is coupled to a memory, a computer program is stored in the memory, and the computer program is configured to execute the above method when executed by the processor.

[0091] This embodiment also provides a computer-readable storage medium on which a computer program is stored. The computer program is used to execute the above method.

[0092] Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. A person with ordinary knowledge in the technical field to which the present application belongs can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be determined by the definition of the claims.

Claims

1. A data processing method based on the TLMP platform, characterized in that: include: Based on the TLMP platform, multiple iterations of machine learning model training are executed sequentially without waiting until the model training is completed; Among them, the several log information generated by each iterative training is stored in the following manner: after each iterative training is completed, the corresponding Log interface is called to store the several log information generated by this iterative training in the message queue in sequence according to the timestamp, and then a write instruction is sent to the Log writing thread to store each log information in the message queue in the database.

2. The data processing method based on the TLMP platform according to claim 1, characterized in that: After calling the corresponding Log interface to sequentially store the plurality of Log information generated by the iterative training into the message queue according to the timestamp, it includes: The log writing thread polls the message queue to sequentially obtain each log information from the message queue and stores it in the database.

3. The data processing method based on the TLMP platform according to claim 1, characterized in that: The storing of each Log information in the message queue in a database includes: Obtain the total number of logs of the plurality of log information in the message queue; When it is determined that the total number of logs is greater than the sum of the maximum processing amounts of all log writing threads, each log writing thread acquires log information several times according to its own maximum processing amount and stores it in the relational database until the remaining log information is less than the sum of the maximum processing amounts of all log writing threads, and the remaining log information is acquired and stored in the relational database using the ratio of the maximum processing amount of the log writing thread to the sum of the maximum processing amounts of all log writing threads as a weight; Otherwise, the Log information is obtained from the message queue and stored in the relational database using the ratio of the maximum processing volume of the Log writing thread to the sum of the maximum processing volumes of all Log writing threads as a weight.

4. The data processing method based on the TLMP platform according to claim 1, characterized in that: The plurality of log information generated by each iterative training is stored in the following manner, including: Evaluating the current iterative training of the machine learning model; The evaluation results are compared with the expected results to obtain the parameter quantity whose deviation between the operating value and the expected value is greater than the deviation threshold, and based on the parameter quantity, the corresponding Log information is retrieved from the database to the visualization front end.

5. The data processing method based on the TLMP platform according to claim 1, characterized in that: The storing of each Log information in the message queue in a database includes: When it is determined based on the unique field in the database that the Log information to be stored is repeated with the existing Log information in the database, discarding the Log information to be stored; Otherwise, the Log information to be stored is inserted into the database.

6. A data processing device based on the TLMP platform, characterized in that: include: An execution module, used to sequentially execute multiple iterations of the machine learning model training without waiting based on the TLMP platform until the model training is completed; The execution module includes the following units to implement the storage of several Log information generated by each iterative training: A first Log storage unit is used to call the corresponding Log interface after each iteration training to store the plurality of Log information generated by the iteration training into the message queue in sequence according to the timestamp; The second Log storage unit is used to send a write instruction to the Log writing thread to store each of the Log information in the message queue in the database.

7. The data processing device based on the TLMP platform according to claim 6, characterized in that: The execution module also includes: The third Log storage unit is used for the Log writing thread to poll the message queue to sequentially obtain each Log information from the message queue and store it in the database.

8. The data processing device based on the TLMP platform according to claim 6, characterized in that: The execution module includes: A statistics acquisition unit, used to acquire the total number of logs of all log information in the message queue; The first judgment unit is used to judge that when the total number of logs is greater than the sum of the maximum processing amounts of all log writing threads, each log writing thread obtains log information several times according to its own maximum processing amount and stores it in the relational database until the remaining log information is less than the sum of the maximum processing amounts of all log writing threads, and then obtains the remaining log information and stores it in the relational database using the ratio of the maximum processing amount of the log writing thread to the sum of the maximum processing amounts of all log writing threads as a weight; Otherwise, the Log information is obtained from the message queue and stored in the relational database using the ratio of the maximum processing volume of the Log writing thread to the sum of the maximum processing volumes of all Log writing threads as a weight.

9. An electronic device, characterized in that it comprises at least one processor, the processor is coupled to a memory, a computer program is stored in the memory, and the computer program is configured to execute the method according to any one of claims 1 to 5 when executed by the processor.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is used to execute the method according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Training data processing method and device and storage medium

    CN111177100A

  • Mass log management method and system

    CN113010483A

  • Capacity adjustment method and device based on log information, equipment and storage medium

    CN114153714A

  • Image classification model incremental training method and system based on Kafka message queue

    CN115880544A