Computing power scheduling engine generation method and electronic device based on programming large model

Abstract

The present application discloses a method and electronic device for generating a computing power scheduling engine based on a programming large model. The method includes: obtaining a target text template, the target text template being filled with multiple template description information of the computing power scheduling engine relative to the computing power scheduling system; converting the multiple template description information in the target text template into a target format file according to a target data exchange format; inputting the target format file into a target programming large model to obtain a scheduling system code for the computing power scheduling system; obtaining a preset scheduling engine code, fusing the scheduling system code with the preset scheduling engine code to obtain a scheduling engine code to be tested; testing the scheduling engine code to be tested to obtain a test result; when the test result is a test pass, generating a target computing power scheduling engine code document for the scheduling engine code to be tested that has passed the test. In an embodiment of the present invention, the manual coding time of developers is reduced and development efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and electronic device for generating a computing power scheduling engine based on a programming large model. Background Art

[0002] Computing scheduling systems have a wide range of applications, playing a vital role in cloud computing, big data processing, artificial intelligence, high-performance computing, and distributed computing. In these areas, computing scheduling systems can improve resource utilization and optimize the efficiency of computing tasks. The integration of computing scheduling systems requires close involvement from development engineers, test engineers, and operations engineers. Development engineers need a deep understanding of the scheduling engine's internal structure, which not only requires a strong technical background but also means they must invest significant time in mastering and integrating the various submodules of the new system when integrating a new system. Because any new integration requires direct involvement and in-depth understanding from engineers, this dependency limits the flexibility and scalability of the scheduling system. Each integration of a new scheduling system increases the development and testing workload significantly, limiting its potential for application in a wider range of scenarios. Summary of the Invention

[0003] The embodiments of the present application provide a method and electronic device for generating a computing power scheduling engine based on a large programming model, which reduces the manual coding time of developers and improves development efficiency.

[0004] In a first aspect, an embodiment of the present application provides a method for generating a computing power scheduling engine based on a large programming model, comprising:

[0005] Obtain a target text template, where the target text template is filled with a plurality of template description information of the computing power scheduling engine relative to the computing power scheduling system;

[0006] Converting the plurality of template description information in the target text template into a target format file according to a target data exchange format;

[0007] Inputting the target format file into a target programming macromodel to obtain a scheduling system code for the computing power scheduling system; wherein the target programming macromodel is obtained by fine-tuning a preset programming macromodel based on a sample annotated file set;

[0008] Obtaining a preset scheduling engine code, and fusing the scheduling system code with the preset scheduling engine code to obtain a scheduling engine code to be tested;

[0009] Testing the scheduling engine code to be tested to obtain a test result;

[0010] When the test result is that the test passes, a target computing power scheduling engine code document is generated for the scheduling engine code to be tested that passes the test.

[0011] In some embodiments, the target programming model is obtained by the following steps:

[0012] Performing data mapping on the sample annotation file set through a preset preprocessing function to obtain a fine-tuning data set;

[0013] Creating fine-tuning parameters, fine-tuning the preset programming model based on the fine-tuning parameters and the fine-tuning data set, and monitoring the fine-tuning process through a preset callback function to obtain a fine-tuning model weight;

[0014] The preset programming large model is evaluated and verified according to the fine-tuning model weights and the preset test file set to obtain the target programming large model.

[0015] In some embodiments, converting the plurality of template description information in the target text template into a target format file according to the target data exchange format includes:

[0016] Performing static detection on the multiple template description information in the target text template by a preset parser to obtain a static detection result;

[0017] When the static detection result is that the test is passed, the template description information that has passed the static detection is translated into the target data exchange format by the parser to obtain a target format file.

[0018] In some embodiments, the testing of the scheduling engine code to be tested to obtain a test result includes:

[0019] Performing a local test on the scheduling engine code to be tested to obtain a first test result;

[0020] Performing a cluster test on the scheduling engine code to be tested to obtain a second test result;

[0021] Performing a deployment test on the scheduling engine code to be tested to obtain a third test result;

[0022] A test result is obtained according to the first test result, the second test result, and the third test result.

[0023] In some embodiments, after testing the scheduling engine code to be tested and obtaining the test results, the method further includes:

[0024] When the test result is that the test fails, optimizing the scheduling engine code to be tested that fails the test to obtain the target scheduling engine code;

[0025] The target scheduling engine code is tested to obtain a target test result.

[0026] In some embodiments, generating a target computing power scheduling engine code document for the scheduling engine code to be tested that has passed the test includes:

[0027] Performing global structural analysis on the tested scheduling engine code through the preset programming model to obtain an element list;

[0028] Performing file analysis on the element list based on a preset deep recursive algorithm and an abstract syntax tree analysis method to obtain an abstract syntax tree;

[0029] Constructing a project tree based on the abstract syntax tree, and performing bidirectional reference relationship analysis on the project tree and the element list to obtain semantic information;

[0030] Generate a target computing power scheduling engine code document based on the project tree and the semantic information.

[0031] In some embodiments, the static detection of the plurality of template description information in the target text template by a preset parser to obtain a static detection result includes:

[0032] Performing mandatory item detection on multiple template description information in the target text template by a preset parser to obtain a first detection result;

[0033] Performing key-value conflict detection on multiple template description information in the target text template by a preset parser to obtain a second detection result;

[0034] Performing an accessibility test on multiple template description information in the target text template using a preset parser to obtain a third test result;

[0035] A static detection result is obtained according to the first detection result, the second detection result, and the third detection result.

[0036] In some embodiments, evaluating and verifying the preset programming large model based on the fine-tuning model weights and a preset test file set to obtain a target programming large model includes:

[0037] Inputting a preset test file set into the preset programming macro model, so that the preset programming macro model evaluates and verifies the test file set based on the fine-tuning model weights to obtain an evaluation result;

[0038] When the evaluation result meets the preset evaluation condition, the preset programming model is determined as the target programming model.

[0039] In a second aspect, an embodiment of the present application provides an electronic device comprising at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a computing power scheduling engine based on a programming large model as described in the first aspect.

[0040] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for generating a computing power scheduling engine based on a programming large model as described in the first aspect.

[0041] The computing power scheduling engine generation method based on the programming big model, electronic device and storage medium provided in the embodiments of the present application have at least the following beneficial effects: first, a target text template filled with multiple template description information of the computing power scheduling engine relative to the computing power scheduling system is obtained, so that detailed description information of the computing power scheduling system can be obtained, which is convenient for subsequent integration and scheduling of various resources; then, according to the target data exchange format, the multiple template description information in the target text template is converted into a target format file to implement demand analysis of multiple template description information; the template description information can be translated into a target format file according to the target data exchange format, so that the template description information is easy to understand and write; thereafter, the target format file is input into the target programming big model to obtain the computing power scheduling system. The scheduling system code is obtained to realize the automatic generation of the scheduling system code, and then the preset scheduling engine code is obtained, and the scheduling system code is integrated with the preset scheduling engine code to realize the integration of the computing power scheduling engine and obtain the scheduling engine code to be tested. After that, the scheduling engine code to be tested is tested to ensure the stability and reliability of the computing power scheduling engine in each link, thereby ensuring the quality and efficiency of the development code, and obtaining the test results, so as to facilitate the subsequent judgment of whether the computing power scheduling engine has achieved automatic integration through the test results. When the test result is a test pass, it means that the scheduling engine code to be tested is completely and high-quality integrated. For the scheduling engine code to be tested that passes the test, a target computing power scheduling engine code document is generated, which is convenient for engineers to read and can quickly locate and fix problems. The embodiment of the present application realizes automatic code generation, automatic testing and document generation of the computing power scheduling system through operations such as demand analysis, code fusion, fusion testing and document generation, accelerates the rapid development and testing of the computing power scheduling system, reduces the manual coding time of developers, and improves development efficiency.

[0042] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a specific method flow chart of a method for generating a computing power scheduling engine based on a programming large model provided by an embodiment of the present application;

[0044] Figure 2 This is a flowchart of a specific method for obtaining a target programming macro model provided by an embodiment of the present application;

[0045] Figure 3 This is a flowchart of a specific method for converting multiple template description information in a target text template into a target format file provided by an embodiment of the present application;

[0046] Figure 4 This is a flowchart of a specific method for testing the scheduling engine code to be tested provided by an embodiment of the present application;

[0047] Figure 5 This is a specific method flow chart of a method for generating a computing power scheduling engine based on a programming large model provided by another embodiment of the present application;

[0048] Figure 6 This is a flowchart of a specific method for generating a target computing power scheduling engine code document provided by an embodiment of the present application;

[0049] Figure 7 This is a flowchart of a specific method for statically detecting multiple template description information in a target text template using a preset parser, provided in one embodiment of the present application;

[0050] Figure 8 This is a flowchart of a specific method for evaluating and verifying a preset programming model based on fine-tuning model weights and a preset test file set, provided in one embodiment of the present application;

[0051] Figure 9 This is a schematic diagram of the structure of the programming model provided by an example of this application;

[0052] Figure 10 This is a schematic diagram of using a large programming model to achieve computing power scheduling fusion provided in an example of this application;

[0053] Figures 11a to 11d is a schematic diagram of different description examples provided in an example of this application;

[0054] Figures 12a to 12d This is a comparison chart of different description examples provided in an example of this application;

[0055] Figure 13 It is a structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0056] This section will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it should not be understood as a limitation on the scope of protection of the present invention.

[0057] In the description of the present invention, if there is a description of first and second, it is only for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0058] Computing scheduling systems have a wide range of applications, playing a vital role in cloud computing, big data processing, artificial intelligence, high-performance computing, and distributed computing. In these areas, computing scheduling systems can improve resource utilization and optimize the efficiency of computing tasks. The integration of computing scheduling systems requires close involvement from development engineers, test engineers, and operations engineers. Development engineers need a deep understanding of the scheduling engine's internal structure, which not only requires a strong technical background but also means they must invest significant time in mastering and integrating the various submodules of the new system when integrating a new system. Because any new integration requires direct involvement and in-depth understanding from engineers, this dependency limits the flexibility and scalability of the scheduling system. Each integration of a new scheduling system increases the development and testing workload significantly, limiting its potential for application in a wider range of scenarios.

[0059] In order to solve the above problems, this embodiment provides a method, electronic device and medium for generating a computing power scheduling engine based on a programming big model, which has at least the following beneficial effects: first, a target text template filled with multiple template description information of the computing power scheduling engine relative to the computing power scheduling system is obtained, so that detailed description information of the computing power scheduling system can be obtained, which is convenient for subsequent integration and scheduling of various resources; then, according to the target data exchange format, the multiple template description information in the target text template is converted into a target format file to realize the demand analysis of multiple template description information; the template description information can be translated into a target format file according to the target data exchange format, so that the template description information is easy to understand and write; then, the target format file is input into the target programming big model to obtain the computing power scheduling system; The scheduling system code of the scheduling system realizes the automatic generation of the scheduling system code, and then obtains the preset scheduling engine code, merges the scheduling system code with the preset scheduling engine code, realizes the fusion of the computing power scheduling engine, and obtains the scheduling engine code to be tested. After that, the scheduling engine code to be tested is tested to ensure the stability and reliability of the computing power scheduling engine in each link, thereby ensuring the quality and efficiency of the development code, and obtaining the test results, so as to facilitate the subsequent judgment of whether the computing power scheduling engine has achieved automatic fusion through the test results. When the test result is a test pass, it means that the scheduling engine code to be tested is complete and high-quality fusion. For the scheduling engine code to be tested that passes the test, a target computing power scheduling engine code document is generated, which is convenient for engineers to read and can quickly locate and fix problems. The embodiment of the present application realizes automatic code generation, automatic testing and document generation of the computing power scheduling system through operations such as demand analysis, code fusion, fusion testing and document generation, accelerates the rapid development and testing of the computing power scheduling system, reduces the manual coding time of developers, and improves development efficiency.

[0060] Reference Figure 1 , Figure 1 This is a specific method flow chart of a method for generating a computing power scheduling engine based on a programming big model provided in one embodiment of the present application. The method for generating a computing power scheduling engine based on a programming big model includes but is not limited to steps S101 to S106.

[0061] Step S101: Obtain a target text template, where the target text template is filled with multiple template description information of the computing power scheduling engine relative to the computing power scheduling system.

[0062] In step S101 of some embodiments, the target text template of the embodiment of the present application is filled with multiple template description information of the computing power scheduling engine relative to the computing power scheduling system. The user fills in multiple template description information through the target text template. The embodiment of the present application first obtains the target text template, so as to obtain the relevant new scheduling system information filled in by the user, which facilitates the subsequent parsing of the target text template to perform code fusion operations.

[0063] It should be noted that the template description information in the embodiment of the present application includes but is not limited to authentication description information, resource description information, image description information and task description information, among which the authentication description information provides the account and password information of the computing power scheduling system; the resource description information provides all resource descriptions of the computing power scheduling system, including name, creation time, computing power type, etc.; the image description information provides the image description of the computing power scheduling system, including name, creation time, computing power type, etc.; the task description information provides the task description of the computing power scheduling system, including name, resource name, image name, center name, etc.

[0064] Step S102: converting the multiple template description information in the target text template into a target format file according to the target data exchange format.

[0065] In step S102 of some embodiments, multiple template description information in the target text template is converted into a target format file according to the target data exchange format, thereby improving the quality and reliability of the target text template, reducing runtime errors, and improving the overall performance of the system and user satisfaction. The template description information that passes the static detection is translated into the target data exchange format through the parser to obtain a target format file, which is convenient for engineers to read and write, can represent dynamic content in the template, is easy to integrate, and can reduce the amount of data transmitted over the network, thereby improving data transmission efficiency.

[0066] Step S103: input the target format file into the target programming big model to obtain the scheduling system code for the computing power scheduling system; wherein, the target programming big model is obtained by fine-tuning the preset programming big model based on the sample annotation file set.

[0067] In step S103 of some embodiments, the target format file is input into the target programming model so that the target programming model automatically generates scheduling system code for the computing power scheduling system based on the target format file, thereby automatically generating code according to demand, helping developers save time and energy.

[0068] It is understandable that the preset programming big model is an artificial intelligence model specifically used for programming tasks. It can understand, generate and optimize code, has powerful code generation capabilities, long sequence modeling capabilities, code modification capabilities, SQL (Structured Query Language) capabilities, and supports dozens of programming languages.

[0069] Step S104: Obtain a preset scheduling engine code, merge the scheduling system code with the preset scheduling engine code, and obtain a scheduling engine code to be tested.

[0070] In step S104 of some embodiments, the preset scheduling engine code is obtained, and the generated scheduling system code is integrated with the preset scheduling engine code, that is, the generated scheduling system code is implanted into the scheduling engine to realize the fusion of computing power scheduling engine, obtain the scheduling engine code to be tested, and realize code fusion.

[0071] Step S105: testing the scheduling engine code to be tested to obtain a test result.

[0072] In step S105 of some embodiments, the scheduling engine code to be tested is tested. Specifically, the scheduling engine code to be tested is tested locally, clustered, and deployed to obtain test results, thereby ensuring that changes to the scheduling engine code to be tested will not destroy existing functions, and at the same time ensuring that a single function or method works as expected, verifying the overall function and performance of the computing power scheduling system, and verifying the performance of the software in a production environment to ensure the security of the software.

[0073] Step S106: When the test result is that the test passes, a target computing power scheduling engine code document is generated for the scheduling engine code to be tested that passes the test.

[0074] In step S106 of some embodiments, when the test result is that the test passes, a code document is automatically generated for the scheduling engine code to be tested through a preset programming model, and the scheduling engine code to be tested is organized into a document that is easy for engineers to read, so that engineers can quickly locate and fix problems.

[0075] Reference Figure 2 , Figure 2 This is a flowchart of a specific method for obtaining a target programming model provided by an embodiment of the present application, and the method includes but is not limited to steps S201 to S203.

[0076] Step S201 : performing data mapping on the sample annotation file set through a preset preprocessing function to obtain a fine-tuning data set.

[0077] Step S202: create fine-tuning parameters, fine-tune the preset programming model based on the fine-tuning parameters and the fine-tuning data set, and monitor the fine-tuning process through a preset callback function to obtain the fine-tuning model weight.

[0078] Step S203 , evaluating and verifying the preset programming large model based on the fine-tuning model weights and the preset test file set to obtain the target programming large model.

[0079] In steps S201 to S203 of some embodiments, the embodiments of the present application first perform data mapping on the sample annotation file set through a preset preprocessing function, map the sample annotation file set into data content suitable for fine-tuning of the preset programming large model, obtain a fine-tuning data set, then create fine-tuning parameters, set the fine-tuning weight file directory, and fine-tune the preset programming large model based on the fine-tuning parameters and the fine-tuning data set to find the model configuration that best suits the current task, and monitor the fine-tuning process through a preset callback function to reduce manual intervention and achieve comprehensive monitoring of the model fine-tuning process to ensure that better model performance is found during the fine-tuning process, obtain the fine-tuning model weight, and save the fine-tuning model weight. After the fine-tuning is completed, use an independent test file set to evaluate the performance of the model, evaluate and verify the preset programming large model based on the fine-tuning model weight and the preset test file set, check the performance of the model on the test data, ensure that the fine-tuning has achieved the expected effect, obtain the target programming large model, and ensure that the model can meet the expected performance standards in actual applications.

[0080] It should be noted that the preprocessing functions in the embodiments of the present application include data set preprocessing functions and single-row data preprocessing functions, wherein the data set preprocessing function is used to process the entire data set, for example, normalizing, standardizing, encoding categorical variables, etc.; the single-row data preprocessing function is used to write preprocessing logic for a single data sample, for example, text cleaning, formatting or feature extraction, etc., and the embodiments of the present application do not impose specific restrictions.

[0081] It is understandable that the fine-tuning parameters include but are not limited to parameters such as learning rate and batch size, and the callback function can be a SwanLab callback function.

[0082] It is worth noting that the sample annotated file set in the embodiments of the present application includes annotated Octopus Scheduling, ModelArts and SLURM (Simple Linux Utility for Resource Management) codes.

[0083] Reference Figure 3 , Figure 3 This is a flowchart of a specific method for converting multiple template description information in a target text template into a target format file provided by an embodiment of the present application. The method includes but is not limited to steps S301 to S302.

[0084] Step S301 : statically detecting multiple template description information in a target text template by using a preset parser to obtain a static detection result.

[0085] Step S302: When the static detection result is that the test is passed, the template description information that has passed the static detection is translated into a target data exchange format by a parser to obtain a target format file.

[0086] In steps S301 to S302 of some embodiments, in the process of converting multiple template description information in the target text template into a target format file, the embodiment of the present application first translates the multiple template description information in the target text template into a target data exchange format through a parser, and is responsible for performing static detection on the target text template. Specifically, the multiple template description information in the target text template is statically detected by a preset parser, wherein the static detection in the embodiment of the present application includes but is not limited to required option filling detection, key value conflict detection, URL (Un iform Resource Locator, Uniform Resource Locator) address accessibility test, etc., to obtain static detection results, thereby improving the quality and reliability of the target text template, reducing runtime errors, and improving the overall performance of the system and user satisfaction. When the static detection result is that the test passes, the template description information that passes the static detection is translated into the target data exchange format through the parser to obtain a target format file, which is convenient for engineers to read and write, can represent dynamic content in the template, is easy to integrate, and can reduce the amount of data transmitted over the network, thereby improving data transmission efficiency.

[0087] It can be understood that the target data exchange format in the embodiment of the present application can be set according to the needs of the user. The target data exchange format in the embodiment of the present application is JSON (JavaScript Object Notation) format. JSON is a lightweight data exchange format that is easy for people to read and write, and is also easy for machines to parse and generate.

[0088] It is worth noting that in the process of translating the template description information that has passed the static detection into the target data exchange format through the parser, the embodiment of the present application will perform demand analysis according to the different types of template description information. Taking the template description information including authentication description information, resource description information, image description information and task description information as an example, in the process of demand analysis, the target text template includes four parts: authentication card, resource description card, image description card, and task description card. The four description cards are translated into the target data exchange format, namely JSON format. The translated target format file mainly includes four main key values, namely authentication, resources, images and tasks, among which the authentication value is a simple dictionary, and the resources, images and tasks values ​​are lists composed of multiple elements, that is, the JSON file after demand analysis corresponds one-to-one to the four description cards, and also includes four parts.

[0089] Reference Figure 4 , Figure 4 This is a flowchart of a specific method for testing the scheduling engine code to be tested provided by an embodiment of the present application, and the method includes but is not limited to steps S401 to S404.

[0090] In some embodiments, after the scheduling system code is integrated with the preset scheduling engine code, the embodiments of the present application will also test the integrated scheduling engine code to be tested. Specifically, the scheduling engine code to be tested is subjected to local testing, cluster testing and deployment testing. The above three tests are tests at three different stages, and each test has its specific purpose and focus. The testing process of the scheduling engine code to be tested is specifically described below.

[0091] Step S401: Perform local testing on the scheduling engine code to be tested to obtain a first test result.

[0092] In step S401 of some embodiments, the scheduling engine code to be tested is locally tested. Local testing is usually performed by developers in a development environment to verify the correctness of the smallest testable unit of the code. Specifically, a unit testing framework (such as JUn it, pytest, NUn it) is used to write test cases, and automated tests are run to ensure that changes to the scheduling engine code to be tested will not destroy existing functions, while ensuring that a single function or method works as expected, and a first test result is obtained. The first test result can be used to quickly discover and fix errors in the code, thereby improving the quality of the scheduling engine code to be tested.

[0093] Step S402: performing a cluster test on the scheduling engine code to be tested to obtain a second test result.

[0094] In step S402 of some embodiments, cluster testing is performed on the scheduling engine code to be tested. Cluster testing occurs when multiple components or services are integrated together, and the purpose is to verify their interaction and collaboration as a whole. Ensure that different services or modules can be correctly integrated and communicated. Detect compatibility issues between interfaces and services, and verify the overall functionality and performance of the system. Specifically, the service is deployed in a simulated production environment, and interface testing is performed using integration testing tools (such as Postman, SoapUI), and end-to-end testing is performed to simulate user operations and business processes to obtain a second test result, so that the overall functionality and performance of the computing power scheduling system can be verified through the second test result.

[0095] Step S403: Perform a deployment test on the scheduling engine code to be tested to obtain a third test result.

[0096] In step S403 of some embodiments, a deployment test is performed on the scheduling engine code to be tested. The deployment test is a test performed in the final stage before the software is ready to be released to the production environment. Specifically, the software is deployed in the pre-production environment, performance testing, load testing and stress testing are performed, and a security test is performed on the scheduling engine code to be tested to obtain a third test result, thereby ensuring the stability and reliability of the software in the production environment, verifying the performance of the software in the production environment, and ensuring the security of the software.

[0097] Step S404: Obtain a test result according to the first test result, the second test result, and the third test result.

[0098] In step S404 of some embodiments, the test results are obtained based on the first test results, the second test results, and the third test results, so that the scheduling engine code to be tested can be tested from different aspects and angles, the code quality can be improved, and the computing power scheduling engine can be further ensured to complete high-quality integration.

[0099] It's important to note that these three testing phases are complementary, each focusing on different aspects of software quality. Local testing focuses on the correctness of code units, cluster testing focuses on the integration between components, and deployment testing focuses on the performance of the software in a production environment. An effective testing strategy should include all of these phases to ensure software quality and user satisfaction.

[0100] Reference Figure 5 , Figure 5 This is a specific method flow chart of a method for generating a computing power scheduling engine based on a programming large model provided by another embodiment of the present application. The method includes but is not limited to steps S501 to S502.

[0101] It should be noted that steps S501 to S502 occur after the scheduling engine code to be tested is tested.

[0102] Step S501: When the test result is that the test fails, the scheduling engine code to be tested that fails the test is optimized to obtain the target scheduling engine code.

[0103] Step S502: Test the target scheduling engine code to obtain a target test result.

[0104] In steps S501 to S502 of some embodiments, after the scheduling engine code to be tested is tested, it may happen that the scheduling engine code to be tested fails the local test, cluster test or deployment test. When the test result is that the test fails, the embodiment of the present application will optimize the scheduling engine code to be tested that fails the test, that is, re-input the scheduling engine code to be tested that fails the test into the preset programming model, regenerate the scheduling engine code through the preset programming model, and merge the preset scheduling engine code with the newly generated scheduling engine code to obtain the target scheduling engine code, and then test the target scheduling engine code, that is, perform local testing, cluster testing and deployment testing on the target scheduling engine code to obtain the target test result, thereby optimizing the scheduling engine code to be tested and improving the readability and maintainability of the code.

[0105] Reference Figure 6 , Figure 6 This is a flowchart of a specific method for generating a target computing power scheduling engine code document provided by an embodiment of the present application, and the method includes but is not limited to steps S601 to S604.

[0106] Step S601 , performing global structural analysis on the tested scheduling engine code through a preset programming macro model to obtain an element list.

[0107] Step S602 : performing file analysis on the element list based on a preset deep recursive algorithm and an abstract syntax tree analysis method to obtain an abstract syntax tree.

[0108] Step S603: construct a project tree based on the abstract syntax tree, and perform bidirectional reference relationship analysis on the project tree and the element list to obtain semantic information.

[0109] Step S604: Generate a target computing power scheduling engine code document based on the project tree and semantic information.

[0110] In steps S601 to S604 of some embodiments, in the process of generating the target computing power scheduling engine code document for the scheduling engine code to be tested that has passed the test, the scheduling engine code to be tested that has passed the test is globally structured parsed through the programming big model. Specifically, the scheduling engine code to be tested is scanned and analyzed to obtain an element list, so that the basic structure and element information of the scheduling engine code to be tested, such as classes, functions, variables, etc., can be determined. Then, the element list is subjected to file analysis based on a preset deep recursive algorithm and an abstract syntax tree analysis method, that is, the element list is deeply recursively encoded, and an abstract syntax tree is constructed at the same time to accurately display the classes, functions, variables and other elements in the scheduling engine code to be tested, so as to realize the display of the hierarchical structure of the scheduling engine code to be tested. Afterwards, a project tree is constructed based on the abstract syntax tree, so that the relationship between each file, class and function in the project can be clearly displayed through the project tree, and the bidirectional reference relationship is parsed for the project tree and the element list. The reference relationship diagram between the code elements is analyzed, including but not limited to function calls, variable references, etc., and the detailed information of the code elements is described to obtain semantic information, and the bidirectional reference relationship between the code objects of the scheduling engine to be tested is parsed. A global contextual perspective is provided for the large model to assist in identifying the functional semantics of the code, and support is provided for generating fine-grained practical code documents. Finally, the target computing power scheduling engine code document is generated based on the project tree and semantic information, and the scheduling engine code to be tested is organized into a document that is easy for engineers to read, so that engineers can quickly locate and fix problems.

[0111] Reference Figure 7 , Figure 7 This is a flowchart of a specific method for statically detecting multiple template description information in a target text template through a preset parser provided in an embodiment of the present application. The method includes but is not limited to steps S701 to S704.

[0112] Step S701: Detect mandatory items of multiple template description information in the target text template using a preset parser to obtain a first detection result.

[0113] Step S702: Perform key-value conflict detection on multiple template description information in the target text template through a preset parser to obtain a second detection result.

[0114] Step S703: Perform an accessibility test on the multiple template description information in the target text template through a preset parser to obtain a third test result.

[0115] Step S704: Obtain a static detection result according to the first detection result, the second detection result, and the third detection result.

[0116] In steps S701 to S704 of some embodiments, in the process of statically detecting multiple template description information in the target text template through a preset parser, the embodiment of the present application detects the required items of multiple template description information in the target text template through a preset parser. Specifically, a template scan is performed on all template description information in the target text template to identify all required tags, and it is checked whether all fields marked as required in the template description information have been filled in, that is, whether the fields corresponding to the required tags are empty are determined, and a first detection result is obtained, thereby realizing the detection of required fields in the target text template and avoiding errors caused by lack of key information in subsequent processing. Leading to erroneous or incomplete processing flow; performing key value conflict detection on multiple template description information in the target text template through a preset parser, specifically, ensuring that each key value of the template description information in the target text template is unique and the value corresponding to each key has no logical conflict, thereby realizing the detection of the key value in the target text template, obtaining a second detection result, and avoiding the situation where the key value in the target text template is repeated or conflicting; performing accessibility test on multiple template description information in the target text template through a preset parser, specifically, performing accessibility test on all URL addresses contained in the template, the embodiment of the present application sends HTTP (Hyper Text Transfer Protocol) requests to these URLs and checks the response status code to confirm whether they can be successfully accessed, thereby obtaining a third detection result, thereby ensuring the validity of each link in the template, avoiding user troubles or errors caused by link failure, and finally, obtaining a static detection result based on the first detection result, the second detection result and the third detection result, thereby improving the quality and reliability of the target text template, reducing runtime errors, and improving the overall performance of the system and user satisfaction.

[0117] Reference Figure 8 , Figure 8 This is a flowchart of a specific method provided by an embodiment of the present application for evaluating and verifying a preset programming large model based on fine-tuning model weights and a preset test file set. The method includes but is not limited to steps S801 to S802.

[0118] Step S801: input a preset test file set into a preset programming macro model, so that the preset programming macro model evaluates and verifies the test file set based on the fine-tuning model weight to obtain an evaluation result.

[0119] Step S802: When the evaluation result satisfies the preset evaluation condition, the preset programming macro model is determined as the target programming macro model.

[0120] In steps S801 to S802 of some embodiments, after the model fine-tuning is completed, the embodiments of the present application use an independent test data set to evaluate the performance of the model. Specifically, the preset test file set is input into the preset programming big model, so that the preset programming big model evaluates and verifies the test file set based on the fine-tuning model weight, checks the performance of the model on the test file set, and obtains an evaluation result. When the evaluation result meets the preset evaluation conditions, it means that the fine-tuned preset programming big model achieves the preset effect. The preset programming big model can be directly determined as the target programming big model to ensure that the performance of the big model on specific tasks is effectively improved and that the fine-tuning achieves the expected effect.

[0121] It should be noted that the preset evaluation conditions in the embodiment of the present application can be set according to the needs of the user, and the embodiment of the present application does not impose specific limitations.

[0122] In order to more clearly illustrate the above-mentioned method of generating a computing power scheduling engine based on a large programming model, a specific example is given below.

[0123] Example 1:

[0124] Example 1 uses the programming big model service to make scheduling engine development simpler and more efficient. The following describes the method for generating a computing power scheduling engine based on the programming big model in this example.

[0125] Reference Figure 9 , Figure 9 This is a structural diagram of a large programming model provided in an example of this application.

[0126] In some embodiments, the programming model consists of three layers: the model base, model fine-tuning, and model serving. The model base includes commonly used models such as CodeQwen, CodeFuse, and MiLM (Mi Language Model). The model typically includes an embedding layer, a multi-layer decoder, and a linear output layer. Each decoder layer includes a self-attention module and a multi-layer perceptron module, as well as input and output layer normalization.

[0127] Fine-tuning large programming models involves datasets, fine-tuning techniques, and monitoring. Datasets include annotated Octopus scheduling, Model Art, and SLURM code implementations. Specifically, fine-tuning techniques include dataset preprocessing, parameter fine-tuning, and inference testing.

[0128] In some embodiments, when fine-tuning a large programming model, first, a dataset preprocessing function and a single-row data preprocessing function are defined based on dataset preprocessing, mapping the original dataset into data content suitable for large model fine-tuning. Then, parameter fine-tuning is performed to create fine-tuning parameters, set the directory for the fine-tuned weight file, and set a SwanLab callback function to monitor the fine-tuning process. After the inference test fine-tuning is completed, the model is evaluated and verified using test data to ensure that the fine-tuning achieved the expected results.

[0129] Understandably, monitoring includes tools like SwanLab, TensorBoard, and Wandb.

[0130] In some embodiments, programming big model services include code generation, code optimization, code testing, and code documentation generation. Automatic code generation refers to the programming big model's ability to automatically generate code based on requirements, saving developers time and effort. Code optimization refers to the programming big model's ability to optimize existing code to improve its readability and maintainability. Code testing refers to the programming big model's ability to automatically generate test cases to help developers identify defects in the code. Code documentation generation refers to the programming big model's ability to automatically generate code documentation to help developers understand the results and functions of the code.

[0131] Example 2:

[0132] Reference Figure 10 , Figure 10 This is a schematic diagram of an example provided in this application for using a large programming model to achieve computing power scheduling fusion.

[0133] This example fine-tunes the large model base and trains it using annotated Octopus Scheduling, Model Arts, and SLURM code datasets to obtain a large model specifically for scheduling system integration. Users are then asked to fill out natural language templates, which the fine-tuned large model parses and generates code for. The scheduling code generated by the large model is then inserted into the original scheduling engine through a packager. After the code is merged, the development tool's testing function allows for rapid local, cluster, and deployment testing of the scheduling engine, ensuring the quality and efficiency of the developed code. Furthermore, the development tool's documentation function organizes the scheduling engine code into easy-to-read documents for engineers, enabling them to quickly locate and fix problems.

[0134] The following describes each step in detail.

[0135] Reference Figures 11a to 11d , Figure 11a This is a schematic diagram of an example authentication description provided in an example of this application; Figure 11b This is a schematic diagram of an example resource description provided in an example of this application; Figure 11cThis is a schematic diagram of an example of image description provided in an example of this application; Figure 11d This is a schematic diagram of a task description example provided in an example of this application.

[0136] In some embodiments, the description of the fusion scheduling system includes authentication description, resource description, image description, and task description. The description is filled in according to the JSON format. Figure 11a , the authentication description provides the Fusion Scheduling System account and password information. Figure 11b , Resource Description provides descriptions of all resources in the fusion scheduling system, including name, creation time, computing power type, etc. Figure 11c , Image description provides the image description of the fusion scheduling system, including name, creation time, computing power type, etc. Figure 11d ,Task description provides the task description of the fusion scheduling system, including name, resource name, image name, center name, etc.

[0137] Reference Figures 12a to 12d , Figure 12a This is a comparison chart of parsing the authentication description provided in an example of this application; Figure 12b This is a comparison chart of parsing resource descriptions provided in an example of this application; Figure 12c This is a comparison chart of parsing image descriptions provided in an example of this application; Figure 12d This is a comparison chart of parsing task descriptions provided in an example of this application.

[0138] After obtaining different descriptions of the fusion scheduling system, the parser in the embodiment of the present application is responsible for translating the natural language template into a JSON file, wherein the natural language template includes four parts: authentication card, resource description card, image description card, and task description card. The parsed JSON file corresponds to each of them and also includes four parts. Figures 12a to 12d The translated format file mainly includes four key values, namely authentication, resources, images, and tasks. The authentication value is a simple dictionary, and the values ​​of resources, images, and tasks are lists consisting of multiple elements. That is, the JSON file after requirement parsing corresponds to the four description cards one by one and also includes four parts.

[0139] It mainly includes four main key values, namely authentication, resources, images and tasks. The authentication value is a simple dictionary, and the resources, images and tasks values ​​are lists composed of multiple elements.

[0140] The parser is also responsible for static detection of the natural language template format, including mandatory field detection, key-value conflict detection, and URL address accessibility testing.

[0141] In some embodiments, the scheduling system programming model API (Application Programming Interface) is integrated into the development environment. The JSON file output in the previous step is input into the programming model to automatically generate the final scheduling system code. The generated code is mainly concentrated in the scheduling system client, with a small amount of code distributed in the service adapter module and launcher.

[0142] After generating the scheduling system code, the code packaging module is responsible for embedding the generated scheduling system code patches into the scheduling engine. In software development, a code patch typically refers to a modification to an existing code base to fix bugs, update functionality, improve performance, or adapt to new development standards. Patches can be minor changes, such as fixing a typo, or major changes, such as refactoring an entire system module.

[0143] Use a version control system to manage patches, ensuring code traceability and rollback. Also, record patch details, including the reason for the change, the scope of impact, and test results. Before applying a patch, conduct a code review to ensure its quality and security. Code patches are an essential part of the software development process; they help maintain the health and vitality of the software.

[0144] After integration is complete, you can perform a one-click test. This includes local testing, cluster testing, and deployment testing, enabling truly automated integration of the computing power scheduling engine.

[0145] Testing is a critical step in ensuring software quality and performance during the software development lifecycle. Local testing, cluster testing, and deployment testing are three different stages of testing, each with its own specific purpose and focus.

[0146] Local testing is typically performed by developers in their development environment to verify the correctness of the smallest testable unit of code. Test cases are written using unit testing frameworks such as JUnit, pytest, and NUnit. Automated tests are run to ensure that code changes do not break existing functionality. This ensures that individual functions or methods work as expected, allowing for quick discovery and fix of code errors, thereby improving code quality.

[0147] Cluster testing occurs when multiple components or services are integrated together to verify their interaction and collaboration as a whole. Ensure that different services or modules can integrate and communicate correctly. Detect compatibility issues between interfaces and services to verify the overall functionality and performance of the system. Deploy services in a simulated production environment. Use integration testing tools (such as Postman and SoapUI) for interface testing. Perform end-to-end testing to simulate user operations and business processes. Cluster testing may require a more complex test environment and configuration. Ensure the isolation and security of test data.

[0148] Deployment testing is the final phase of software development before it is released to production. It aims to ensure the stability and reliability of the software in production. It verifies the software's performance in production. The software is deployed in a pre-production environment. Performance, load, and stress testing are performed, along with security testing, to ensure software security.

[0149] These three testing phases are complementary, each focusing on a different aspect of software quality. Local testing focuses on the correctness of code units, cluster testing focuses on the integration between components, and deployment testing focuses on the performance of the software in a production environment. An effective testing strategy should include all of these phases to ensure software quality and user satisfaction.

[0150] Through local testing, cluster testing, and deployment testing, the computing power scheduling engine achieves high-quality integration.

[0151] After code optimization is complete, the programming model automatically generates code documentation. Using global structural parsing technology, the model uses deep recursion and abstract syntax tree analysis to accurately parse classes and functions within each file and build a project tree. It also resolves bidirectional references between code objects, providing the model with a global contextual perspective to assist in identifying the functional semantics of the code and supporting the generation of fine-grained, practical code documentation.

[0152] It should be noted that during the document generation phase, the large model generates fine-grained code documentation, which includes the function description, parameter description, code description, precautions and usage examples of the code object.

[0153] In some embodiments, the embodiments of the present application can simplify the interface integration process of different sub-scheduling systems. By leveraging the advantages of the programming large model to automatically generate code and automatically generate documents based on the learning of the demand template, development engineers can complete the rapid integration of the scheduling system without having to deeply understand the distribution and functions of all sub-modules of the scheduling engine, making the development process more efficient, the test coverage more comprehensive, and improving the maintainability of the system. This method significantly reduces the development and testing workload and provides an efficient solution for the multi-system integration of the computing power scheduling engine.

[0154] Reference Figure 13 , taking the example that the control processor 1001 and the memory 1002 in the electronic device 1000 can be connected via a bus. The memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk memory, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the control processor 1001, and these remote memories may be connected to the electronic device 1000 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0155] Those skilled in the art will understand that Figure 13 The device structure shown in the figure does not constitute a limitation on the electronic device 1000, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0156] The embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, which are executed by one or more control processors, for example, Figure 13 One of the control processors 1001 executes, which can enable the above-mentioned one or more control processors to execute the computing power scheduling engine generation method based on the programming large model in the above-mentioned method embodiment.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0158] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A method for generating a computing power scheduling engine based on a large programming model, characterized in that: include: Obtain a target text template, where the target text template is filled with a plurality of template description information of the computing power scheduling engine relative to the computing power scheduling system; Converting the plurality of template description information in the target text template into a target format file according to a target data exchange format; the template description information includes authentication description information, resource description information, image description information, and task description information; Inputting the target format file into a target programming model to obtain a scheduling system code for the computing power scheduling system; Obtaining a preset scheduling engine code, and fusing the scheduling system code with the preset scheduling engine code to obtain a scheduling engine code to be tested; Testing the scheduling engine code to be tested to obtain a test result; When the test result is that the test passes, a target computing power scheduling engine code document is generated for the scheduling engine code to be tested that passes the test; The target programming model is obtained by the following steps: Perform data mapping on the sample annotation file set through a preset preprocessing function to obtain a fine-tuning dataset; Creating fine-tuning parameters, fine-tuning the preset programming model based on the fine-tuning parameters and the fine-tuning data set, and monitoring the fine-tuning process through a preset callback function to obtain a fine-tuning model weight; Evaluating and verifying the preset programming model based on the fine-tuning model weights and the preset test file set to obtain a target programming model; The converting the plurality of template description information in the target text template into a target format file according to the target data exchange format includes: Performing static detection on the multiple template description information in the target text template by a preset parser to obtain a static detection result; When the static detection result is that the test is passed, the template description information that has passed the static detection is translated into the target data exchange format by the parser to obtain a target format file.

2. The method for generating a computing power scheduling engine based on a programming large model according to claim 1, characterized in that: The testing of the scheduling engine code to be tested to obtain a test result includes: Performing a local test on the scheduling engine code to be tested to obtain a first test result; Performing a cluster test on the scheduling engine code to be tested to obtain a second test result; Performing a deployment test on the scheduling engine code to be tested to obtain a third test result; A test result is obtained according to the first test result, the second test result, and the third test result.

3. The method for generating a computing power scheduling engine based on a programming large model according to claim 1, characterized in that: After testing the scheduling engine code to be tested and obtaining the test results, the method further includes: When the test result is that the test fails, optimizing the scheduling engine code to be tested that fails the test to obtain the target scheduling engine code; The target scheduling engine code is tested to obtain a target test result.

4. The method for generating a computing power scheduling engine based on a programming large model according to claim 1, characterized in that: The target computing power scheduling engine code document is generated for the scheduling engine code to be tested that has passed the test, including: Performing global structural analysis on the tested scheduling engine code through the preset programming model to obtain an element list; Performing file analysis on the element list based on a preset deep recursive algorithm and an abstract syntax tree analysis method to obtain an abstract syntax tree; Constructing a project tree based on the abstract syntax tree, and performing bidirectional reference relationship analysis on the project tree and the element list to obtain semantic information; Generate a target computing power scheduling engine code document based on the project tree and the semantic information.

5. The method for generating a computing power scheduling engine based on a programming large model according to claim 1, characterized in that: The static detection of the plurality of template description information in the target text template by a preset parser is performed to obtain a static detection result, including: Performing mandatory item detection on multiple template description information in the target text template by a preset parser to obtain a first detection result; Performing key-value conflict detection on multiple template description information in the target text template by a preset parser to obtain a second detection result; Performing an accessibility test on multiple template description information in the target text template using a preset parser to obtain a third test result; A static detection result is obtained according to the first detection result, the second detection result, and the third detection result.

6. The method for generating a computing power scheduling engine based on a programming large model according to claim 1, characterized in that: The step of evaluating and verifying the preset programming model based on the fine-tuning model weights and the preset test file set to obtain a target programming model includes: Inputting a preset test file set into the preset programming macro model, so that the preset programming macro model evaluates and verifies the test file set based on the fine-tuning model weights to obtain an evaluation result; When the evaluation result meets the preset evaluation condition, the preset programming model is determined as the target programming model.

7. An electronic device, characterized in that: It includes at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the computing power scheduling engine generation method based on the programming large model as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for generating a computing power scheduling engine based on a programming large model as described in any one of claims 1 to 6.

Images