Computing power scheduling engine generation method based on large programming model and electronic equipment

Abstract

The invention discloses a computing power scheduling engine generation method based on a large programming model and electronic equipment, and the method comprises the steps: obtaining a target text template which is filled with a plurality of pieces of template description information of a computing power scheduling engine relative to a computing power scheduling system; according to the target data exchange format, converting multiple pieces of template description information in the target text template into a target format file; inputting the target format file into the target programming large model to obtain a scheduling system code for the computing power scheduling system; obtaining a preset scheduling engine code, and fusing the scheduling system code with the preset scheduling engine code to obtain a to-be-tested scheduling engine code; testing the to-be-tested scheduling engine code to obtain a test result; and when the test result is that the test is passed, generating a target computing power scheduling engine code document for the to-be-tested scheduling engine code passing the test. In the embodiment of the invention, the manual coding time of developers is reduced, and the development efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and electronic device for generating a computing power scheduling engine based on a programming large model. Background Art

[0002] The computing power scheduling system has a wide range of applications. It plays an important role in cloud computing, big data processing, artificial intelligence, high-performance computing, and distributed computing. In these fields, the computing power scheduling system can improve resource utilization and optimize the execution efficiency of computing tasks. The integration of computing power scheduling systems requires the close participation of development engineers, test engineers, and operation and maintenance engineers. Development engineers need to have an in-depth understanding of the internal structure of the scheduling engine, which not only requires them to have a deep technical background, but also means that when integrating new systems, they must invest a lot of time to master and integrate the various sub-modules of the new system. Since any new integration requires the direct participation and in-depth understanding of engineers, this dependence limits the flexibility and scalability of the scheduling system, resulting in a large increase in development and testing workload for each integration of a new scheduling system, which limits its application potential in a wider range of scenarios. Summary of the invention

[0003] The embodiments of the present application provide a method and electronic device for generating a computing power scheduling engine based on a large programming model, which reduces the manual coding time of developers and improves development efficiency.

[0004] In a first aspect, an embodiment of the present application provides a method for generating a computing power scheduling engine based on a programming large model, including:

[0005] Obtain a target text template, where the target text template is filled with a plurality of template description information of the computing power scheduling engine relative to the computing power scheduling system;

[0006] According to a target data exchange format, converting the plurality of template description information in the target text template into a target format file;

[0007] Inputting the target format file into the target programming big model to obtain the scheduling system code for the computing power scheduling system; wherein the target programming big model is obtained by fine-tuning the preset programming big model based on the sample annotated file set;

[0008] Obtaining a preset scheduling engine code, and fusing the scheduling system code with the preset scheduling engine code to obtain a scheduling engine code to be tested;

[0009] Testing the scheduling engine code to be tested to obtain a test result;

[0010] When the test result is that the test passes, a target computing power scheduling engine code document is generated for the scheduling engine code to be tested that passes the test.

[0011] In some embodiments, the target programming model is obtained by the following steps:

[0012] Performing data mapping on the sample annotation file set through a preset preprocessing function to obtain a fine-tuning data set;

[0013] Creating fine-tuning parameters, fine-tuning the preset programming model based on the fine-tuning parameters and the fine-tuning data set, and monitoring the fine-tuning process through a preset callback function to obtain a fine-tuning model weight;

[0014] The preset programming large model is evaluated and verified according to the fine-tuning model weights and the preset test file set to obtain the target programming large model.

[0015] In some embodiments, converting the plurality of template description information in the target text template into a target format file according to the target data exchange format includes:

[0016] Performing static detection on multiple template description information in the target text template through a preset parser to obtain a static detection result;

[0017] When the static detection result is that the test is passed, the template description information that has passed the static detection is translated into the target data exchange format through the parser to obtain a target format file.

[0018] In some embodiments, the testing of the scheduling engine code to be tested to obtain a test result includes:

[0019] Performing a local test on the scheduling engine code to be tested to obtain a first test result;

[0020] Performing a cluster test on the scheduling engine code to be tested to obtain a second test result;

[0021] Performing a deployment test on the scheduling engine code to be tested to obtain a third test result;

[0022] A test result is obtained according to the first test result, the second test result and the third test result.

[0023] In some embodiments, after testing the scheduling engine code to be tested and obtaining the test result, the method further includes:

[0024] When the test result is that the test fails, optimizing the scheduling engine code to be tested that fails the test to obtain a target scheduling engine code;

[0025] The target scheduling engine code is tested to obtain a target test result.

[0026] In some embodiments, the generating of the target computing power scheduling engine code document for the scheduling engine code to be tested that has passed the test includes:

[0027] Performing global structural analysis on the tested scheduling engine code through the preset programming macro model to obtain an element list;

[0028] Performing file analysis on the element list based on a preset deep recursive algorithm and an abstract syntax tree analysis method to obtain an abstract syntax tree;

[0029] Constructing a project tree based on the abstract syntax tree, and performing bidirectional reference relationship analysis on the project tree and the element list to obtain semantic information;

[0030] Generate a target computing power scheduling engine code document based on the project tree and the semantic information.

[0031] In some embodiments, the static detection of the plurality of template description information in the target text template by a preset parser to obtain a static detection result includes:

[0032] Performing mandatory item detection on multiple template description information in the target text template through a preset parser to obtain a first detection result;

[0033] Performing key-value conflict detection on multiple template description information in the target text template through a preset parser to obtain a second detection result;

[0034] Performing an accessibility test on multiple template description information in the target text template through a preset parser to obtain a third test result;

[0035] A static detection result is obtained according to the first detection result, the second detection result and the third detection result.

[0036] In some embodiments, the step of evaluating and verifying the preset programming large model according to the fine-tuning model weights and the preset test file set to obtain the target programming large model includes:

[0037] Inputting a preset test file set into the preset programming big model, so that the preset programming big model evaluates and verifies the test file set based on the fine-tuning model weights to obtain an evaluation result;

[0038] When the evaluation result meets the preset evaluation condition, the preset programming big model is determined as the target programming big model.

[0039] In a second aspect, an embodiment of the present application provides an electronic device comprising at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a computing power scheduling engine based on a programming large model as described in the first aspect.

[0040] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for generating a computing power scheduling engine based on a programming big model as described in the first aspect.

[0041] The computing power scheduling engine generation method based on the programming big model, electronic device and storage medium provided in the embodiments of the present application have at least the following beneficial effects: first, a target text template filled with multiple template description information of the computing power scheduling engine relative to the computing power scheduling system is obtained, so that detailed description information of the computing power scheduling system can be obtained, which is convenient for subsequent integration and scheduling of various resources; then, according to the target data exchange format, the multiple template description information in the target text template is converted into a target format file, and the demand analysis of multiple template description information is realized, and the template description information can be translated into a target format file according to the target data exchange format, so that the template description information is easy to understand and write; then, the target format file is input into the target programming big model to obtain the computing power scheduling system. The scheduling system code is obtained to realize automatic code generation of the scheduling system, and then the preset scheduling engine code is obtained, and the scheduling system code is integrated with the preset scheduling engine code to realize the integration of the computing power scheduling engine, and the scheduling engine code to be tested is obtained. After that, the scheduling engine code to be tested is tested to ensure the stability and reliability of the computing power scheduling engine in each link, thereby ensuring the quality and efficiency of the development code, and obtaining the test results, so as to facilitate the subsequent judgment of whether the computing power scheduling engine has achieved automatic integration through the test results. When the test result is that the test passes, it means that the scheduling engine code to be tested is completely and high-quality integrated. For the scheduling engine code to be tested that passes the test, a target computing power scheduling engine code document is generated, which is convenient for engineers to read and can quickly locate and fix problems. The embodiment of the present application realizes automatic code generation, automatic testing and document generation of the computing power scheduling system through operations such as demand analysis, code fusion, fusion testing and document generation, accelerates the rapid development and testing of the computing power scheduling system, reduces the manual coding time of developers, and improves development efficiency.

[0042] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a specific method flow chart of a method for generating a computing power scheduling engine based on a programming large model provided in an embodiment of the present application;

[0044] Figure 2 It is a flowchart of a specific method for obtaining a target programming macro model provided by an embodiment of the present application;

[0045] Figure 3 It is a flowchart of a specific method for converting multiple template description information in a target text template into a target format file provided by an embodiment of the present application;

[0046] Figure 4 This is a flowchart of a specific method for testing the scheduling engine code to be tested provided by an embodiment of the present application;

[0047] Figure 5 It is a specific method flow chart of a method for generating a computing power scheduling engine based on a programming large model provided by another embodiment of the present application;

[0048] Figure 6 It is a flowchart of a specific method for generating a target computing power scheduling engine code document provided by an embodiment of the present application;

[0049] Figure 7 It is a flowchart of a specific method for statically detecting multiple template description information in a target text template by a preset parser provided in an embodiment of the present application;

[0050] Figure 8 It is a flowchart of a specific method for evaluating and verifying a preset programming large model according to fine-tuning model weights and a preset test file set provided in an embodiment of the present application;

[0051] Fig. 9 It is a structural diagram of a programming model provided by an example of this application;

[0052] Fig.10 This is a schematic diagram of using a programming big model to achieve computing power scheduling fusion provided by an example of this application;

[0053] Figures 11a to 11d is a schematic diagram of different description examples provided in an example of this application;

[0054] Figures 12a to 12d is a comparison chart of different description examples provided in an example of this application;

[0055] Fig.13 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0056] This section will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it cannot be understood as a limitation on the scope of protection of the present invention.

[0057] In the description of the present invention, if there is a description of first and second, it is only for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0058] The computing power scheduling system has a wide range of applications. It plays an important role in cloud computing, big data processing, artificial intelligence, high-performance computing, and distributed computing. In these fields, the computing power scheduling system can improve resource utilization and optimize the execution efficiency of computing tasks. The integration of computing power scheduling systems requires the close participation of development engineers, test engineers, and operation and maintenance engineers. Development engineers need to have an in-depth understanding of the internal structure of the scheduling engine, which not only requires them to have a deep technical background, but also means that when integrating new systems, they must invest a lot of time to master and integrate the various sub-modules of the new system. Since any new integration requires the direct participation and in-depth understanding of engineers, this dependence limits the flexibility and scalability of the scheduling system, resulting in a large increase in development and testing workload for each integration of a new scheduling system, which limits its application potential in a wider range of scenarios.

[0059] In order to solve the above problems, this embodiment provides a computing power scheduling engine generation method, electronic device and medium based on a programming big model, which has at least the following beneficial effects: first, a target text template filled with multiple template description information of the computing power scheduling engine relative to the computing power scheduling system is obtained, so that a detailed description information of the computing power scheduling system can be obtained, which is convenient for the subsequent integration and scheduling of various resources, and then the multiple template description information in the target text template is converted into a target format file according to the target data exchange format to realize the demand analysis of multiple template description information, and the template description information can be translated into a target format file according to the target data exchange format, so that the template description information is easy to understand and write, and then the target format file is input into the target programming big model to obtain the computing power scheduling system. The scheduling system code of the scheduling system realizes the automatic code generation of the scheduling system, and then obtains the preset scheduling engine code, merges the scheduling system code with the preset scheduling engine code, realizes the fusion of the computing power scheduling engine, and obtains the scheduling engine code to be tested. After that, the scheduling engine code to be tested is tested to ensure the stability and reliability of the computing power scheduling engine in each link, thereby ensuring the quality and efficiency of the development code, and obtaining the test results, so as to facilitate the subsequent judgment of whether the computing power scheduling engine has achieved automatic fusion through the test results. When the test result is a test pass, it means that the scheduling engine code to be tested is completely and high-quality integrated. For the scheduling engine code to be tested that has passed the test, a target computing power scheduling engine code document is generated, which is convenient for engineers to read and can quickly locate and fix problems. The embodiment of the present application realizes automatic code generation, automatic testing and document generation of the computing power scheduling system through operations such as demand analysis, code fusion, fusion testing and document generation, accelerates the rapid development and testing of the computing power scheduling system, reduces the manual coding time of developers, and improves development efficiency.

[0060] Reference Figure 1 , Figure 1 It is a specific method flow chart of a computing power scheduling engine generation method based on a programming big model provided in one embodiment of the present application. The computing power scheduling engine generation method based on a programming big model includes but is not limited to steps S101 to S106.

[0061] Step S101, obtaining a target text template, where the target text template is filled with a plurality of template description information of the computing power scheduling engine relative to the computing power scheduling system.

[0062] In step S101 of some embodiments, the target text template of the embodiment of the present application is filled with multiple template description information of the computing power scheduling engine relative to the computing power scheduling system. The user fills in multiple template description information through the target text template. The embodiment of the present application first obtains the target text template, so as to obtain the relevant new scheduling system information filled in by the user, which facilitates the subsequent parsing of the target text template to perform code fusion operations.

[0063] It should be noted that the template description information in the embodiment of the present application includes but is not limited to authentication description information, resource description information, image description information and task description information, wherein the authentication description information provides the account and password information of the computing power scheduling system; the resource description information provides all resource descriptions of the computing power scheduling system, including name, creation time, computing power type, etc.; the image description information provides the image description of the computing power scheduling system, including name, creation time, computing power type, etc.; the task description information provides the task description of the computing power scheduling system, including name, resource name, image name, center name, etc.

[0064] Step S102: converting the multiple template description information in the target text template into a target format file according to the target data exchange format.

[0065] In step S102 of some embodiments, multiple template description information in the target text template is converted into a target format file in accordance with the target data exchange format, thereby improving the quality and reliability of the target text template, reducing runtime errors, and improving the overall performance of the system and user satisfaction. The template description information that passes the static detection is translated into the target data exchange format through a parser to obtain a target format file, which is convenient for engineers to read and write, can represent dynamic content in the template, is easy to integrate, and can reduce the amount of data transmitted over the network and improve data transmission efficiency.

[0066] Step S103, input the target format file into the target programming big model to obtain the scheduling system code for the computing power scheduling system; wherein, the target programming big model is obtained by fine-tuning the preset programming big model based on the sample annotation file set.

[0067] In step S103 of some embodiments, the target format file is input into the target programming model, so that the target programming model automatically generates scheduling system code for the computing power scheduling system according to the target format file, thereby automatically generating code according to demand, helping developers save time and energy.

[0068] It can be understood that the preset programming big model is an artificial intelligence model specifically used for programming tasks. It can understand, generate and optimize code, has powerful code generation capabilities, long sequence modeling capabilities, code modification capabilities, SQL (Structured Query Language) capabilities, and supports dozens of programming languages.

[0069] Step S104, obtaining a preset scheduling engine code, and integrating the scheduling system code with the preset scheduling engine code to obtain a scheduling engine code to be tested.

[0070] In step S104 of some embodiments, the preset scheduling engine code is obtained, and the generated scheduling system code is merged with the preset scheduling engine code, that is, the generated scheduling system code is implanted into the scheduling engine to realize the fusion of computing power scheduling engine, obtain the scheduling engine code to be tested, and realize the fusion of code.

[0071] Step S105, testing the scheduling engine code to be tested to obtain a test result.

[0072] In step S105 of some embodiments, the scheduling engine code to be tested is tested. Specifically, the scheduling engine code to be tested is tested locally, clustered, and deployed to obtain test results, thereby ensuring that changes to the scheduling engine code to be tested will not destroy existing functions, while ensuring that a single function or method works as expected, verifying the overall functionality and performance of the computing power scheduling system, and verifying the performance of the software in a production environment to ensure the security of the software.

[0073] Step S106: When the test result is that the test passes, a target computing power scheduling engine code document is generated for the scheduling engine code to be tested that passes the test.

[0074] In step S106 of some embodiments, when the test result is that the test passes, a code document is automatically generated for the scheduling engine code to be tested that passes the test through a preset programming model, and the scheduling engine code to be tested is organized into a document that is easy for engineers to read, so that engineers can quickly locate and fix problems.

[0075] Reference Figure 2 , Figure 2 This is a flowchart of a specific method for obtaining a target programming model provided by an embodiment of the present application, and the method includes but is not limited to steps S201 to S203.

[0076] Step S201 , performing data mapping on a sample annotation file set through a preset preprocessing function to obtain a fine-tuning data set.

[0077] Step S202, creating fine-tuning parameters, fine-tuning the preset programming model based on the fine-tuning parameters and the fine-tuning data set, and monitoring the fine-tuning process through a preset callback function to obtain the fine-tuning model weight.

[0078] Step S203, evaluating and verifying the preset programming large model according to the fine-tuning model weights and the preset test file set to obtain the target programming large model.

[0079] In steps S201 to S203 of some embodiments, the embodiments of the present application first perform data mapping on the sample annotation file set through a preset preprocessing function, map the sample annotation file set into data content suitable for fine-tuning of the preset programming large model, obtain a fine-tuning data set, then create fine-tuning parameters, set the weight file directory after fine-tuning, and fine-tune the preset programming large model based on the fine-tuning parameters and the fine-tuning data set to find the model configuration that best suits the current task, and monitor the fine-tuning process through a preset callback function to reduce manual intervention, and achieve comprehensive monitoring of the model fine-tuning process to ensure that better model performance is found during the fine-tuning process, obtain the fine-tuning model weight, and save the fine-tuning model weight. After the fine-tuning is completed, use an independent test file set to evaluate the performance of the model, evaluate and verify the preset programming large model according to the fine-tuning model weight and the preset test file set, check the performance of the model on the test data, ensure that the fine-tuning has achieved the expected effect, obtain the target programming large model, and ensure that the model can meet the expected performance standards in actual applications.

[0080] It should be noted that the preprocessing functions in the embodiments of the present application include data set preprocessing functions and single-row data preprocessing functions, wherein the data set preprocessing function is used to process the entire data set, for example, normalizing, standardizing, encoding categorical variables, etc.; the single-row data preprocessing function is used to write preprocessing logic for a single data sample, for example, text cleaning, formatting or feature extraction, etc., and the embodiments of the present application do not make specific restrictions.

[0081] It is understandable that the fine-tuning parameters include but are not limited to parameters such as learning rate and batch size, and the callback function can be a SwanLab callback function.

[0082] It is worth noting that the sample annotated file set in the embodiment of the present application includes annotated Octopus Scheduling, ModelArts and SLURM (Simple Linux Utility for Resource Management) codes.

[0083] Reference Figure 3 , Figure 3 It is a flowchart of a specific method for converting multiple template description information in a target text template into a target format file provided by an embodiment of the present application. The method includes but is not limited to steps S301 to S302.

[0084] Step S301, statically detecting multiple template description information in a target text template through a preset parser to obtain a static detection result.

[0085] Step S302, when the static detection result is that the test is passed, the template description information that passes the static detection is translated into the target data exchange format by the parser to obtain the target format file.

[0086] In steps S301 to S302 of some embodiments, in the process of converting multiple template description information in the target text template into a target format file, the embodiment of the present application first translates the multiple template description information in the target text template into a target data exchange format through a parser, and is responsible for statically detecting the target text template. Specifically, a preset parser is used to statically detect the multiple template description information in the target text template, wherein the static detection in the embodiment of the present application includes but is not limited to required option filling detection, key value conflict detection, URL (Uniform Resource Locator) address accessibility test, etc., to obtain a static detection result, thereby improving the quality and reliability of the target text template, reducing runtime errors, and improving the overall performance of the system and user satisfaction. When the static detection result is that the test passes, the template description information that passes the static detection is translated into the target data exchange format through the parser to obtain a target format file, which is convenient for engineers to read and write, can represent dynamic content in the template, is easy to integrate, and can reduce the amount of data transmitted over the network, thereby improving data transmission efficiency.

[0087] It can be understood that the target data exchange format in the embodiment of the present application can be set according to the needs of the user. The target data exchange format in the embodiment of the present application is JSON (JavaScript Object Notation) format. JSON is a lightweight data exchange format that is easy for people to read and write, and is also easy for machines to parse and generate.

[0088] It is worth noting that in the process of translating the template description information that has passed the static detection into the target data exchange format through the parser, the embodiment of the present application will perform demand analysis according to the different types of template description information. Taking the template description information including authentication description information, resource description information, image description information and task description information as an example, in the process of demand analysis, the target text template includes four parts: authentication card, resource description card, image description card and task description card. The four description cards are translated into the target data exchange format, namely, JSON format. The translated target format file mainly includes four main key values, namely, authentication, resources, images and tasks, among which the authentication value is a simple dictionary, and the resources, images and tasks values ​​are lists composed of multiple elements, that is, the JSON file after the demand analysis corresponds one by one to the four description cards and also includes four parts.

[0089] Reference Figure 4 , Figure 4 This is a flowchart of a specific method for testing the scheduling engine code to be tested provided by an embodiment of the present application, and the method is not limited to steps S401 to S404.

[0090] In some embodiments, after the scheduling system code is merged with the preset scheduling engine code, the embodiments of the present application will also test the merged scheduling engine code to be tested. Specifically, the scheduling engine code to be tested is subjected to local testing, cluster testing and deployment testing. The above three tests are tests at three different stages, and each test has its specific purpose and focus. The testing process of the scheduling engine code to be tested is specifically described below.

[0091] Step S401: Perform local testing on the scheduling engine code to be tested to obtain a first test result.

[0092] In step S401 of some embodiments, the scheduling engine code to be tested is locally tested. The local test is usually performed by the developer in the development environment to verify the correctness of the smallest testable unit of the code. Specifically, a unit testing framework (such as JUn it, pytest, NUn it) is used to write test cases, and automated tests are run to ensure that changes to the scheduling engine code to be tested will not destroy existing functions, while ensuring that a single function or method works as expected, and a first test result is obtained. The first test result can be used to quickly discover and fix errors in the code, thereby improving the quality of the scheduling engine code to be tested.

[0093] Step S402: performing a cluster test on the scheduling engine code to be tested to obtain a second test result.

[0094] In step S402 of some embodiments, cluster testing is performed on the scheduling engine code to be tested. Cluster testing occurs when multiple components or services are integrated together, and the purpose is to verify their interaction and collaboration as a whole. Ensure that different services or modules can be correctly integrated and communicated. Detect compatibility issues between interfaces and services, and verify the overall functionality and performance of the system. Specifically, the service is deployed in a simulated production environment, and interface testing is performed using integrated testing tools (such as Postman, SoapUI), end-to-end testing is performed, user operations and business processes are simulated, and a second test result is obtained, so that the overall functionality and performance of the computing power scheduling system can be verified through the second test result.

[0095] Step S403: perform deployment test on the scheduling engine code to be tested to obtain a third test result.

[0096] In step S403 of some embodiments, a deployment test is performed on the scheduling engine code to be tested. The deployment test is a test performed in the final stage before the software is ready to be released to the production environment. Specifically, the software is deployed in a pre-production environment, performance testing, load testing, and stress testing are performed, and a security test is performed on the scheduling engine code to be tested to obtain a third test result, thereby ensuring the stability and reliability of the software in the production environment, being able to verify the performance of the software in the production environment, and ensuring the security of the software.

[0097] Step S404, obtaining a test result according to the first test result, the second test result and the third test result.

[0098] In step S404 of some embodiments, the test result is obtained according to the first test result, the second test result and the third test result, so that the scheduling engine code to be tested can be tested from different aspects and angles, the code quality can be improved, and the computing power scheduling engine can be further ensured to complete high-quality integration.

[0099] It should be noted that these three testing phases are complementary, and each phase focuses on different aspects of software quality. Local testing focuses on the correctness of code units, cluster testing focuses on the integration between components, and deployment testing focuses on the performance of the software in the production environment. An effective testing strategy should include all of these phases to ensure software quality and user satisfaction.

[0100] Reference Figure 5 , Figure 5 It is a specific method flow chart of a method for generating a computing power scheduling engine based on a programming big model provided in another embodiment of the present application, and the method includes but is not limited to steps S501 to S502.

[0101] It should be noted that step S501 to step S502 occur after the scheduling engine code to be tested is tested.

[0102] Step S501, when the test result is that the test fails, the scheduling engine code to be tested that fails the test is optimized to obtain the target scheduling engine code.

[0103] Step S502, testing the target scheduling engine code to obtain a target test result.

[0104] In steps S501 to S502 of some embodiments, after the scheduling engine code to be tested is tested, it may happen that the scheduling engine code to be tested fails the local test, cluster test or deployment test. When the test result is that the test fails, the embodiment of the present application will optimize the scheduling engine code to be tested that fails the test, that is, re-input the scheduling engine code to be tested that fails the test into the preset programming big model, regenerate the scheduling engine code through the preset programming big model, and merge the preset scheduling engine code with the newly generated scheduling engine code to obtain the target scheduling engine code, and then test the target scheduling engine code, that is, perform local testing, cluster testing and deployment testing on the target scheduling engine code to obtain the target test result, thereby optimizing the scheduling engine code to be tested and improving the readability and maintainability of the code.

[0105] Reference Figure 6 , Figure 6 It is a flowchart of a specific method for generating a target computing power scheduling engine code document provided by an embodiment of the present application, and the method includes but is not limited to steps S601 to S604.

[0106] Step S601, performing global structural analysis on the tested scheduling engine code through a preset programming macro model to obtain an element list.

[0107] Step S602: Perform file analysis on the element list based on a preset deep recursive algorithm and an abstract syntax tree analysis method to obtain an abstract syntax tree.

[0108] Step S603, constructing a project tree based on the abstract syntax tree, and performing bidirectional reference relationship analysis on the project tree and the element list to obtain semantic information.

[0109] Step S604: Generate a target computing power scheduling engine code document based on the project tree and semantic information.

[0110] In steps S601 to S604 of some embodiments, in the process of generating a target computing power scheduling engine code document for the scheduling engine code to be tested that has passed the test, a global structural analysis is performed on the scheduling engine code to be tested that has passed the test through a programming large model. Specifically, the scheduling engine code to be tested is scanned and analyzed to obtain an element list, so that the basic structure and element information of the scheduling engine code to be tested, such as classes, functions, variables, etc., can be determined. Then, based on a preset deep recursive algorithm and an abstract syntax tree analysis method, a file analysis is performed on the element list, that is, the element list is deeply recursively encoded, and an abstract syntax tree is constructed at the same time, so as to accurately display the classes, functions, variables and other elements in the scheduling engine code to be tested, so as to realize the display of the hierarchical structure of the scheduling engine code to be tested. ,Afterwards, a project tree is constructed based on the abstract syntax tree, so that the ,relationships between the files, classes and functions in the project can be clearly displayed through the ,project tree, and the bidirectional reference relationship is parsed for the project tree and the ,element list, and the reference relationship diagram between the code elements is ,analyzed, including but not limited to function calls, variable references, etc., and the detailed information of the code elements is ,described to obtain semantic information, and the bidirectional reference relationship between the ,code objects of the scheduling engine to be tested is parsed, providing a global context perspective for the ,large model to assist in identifying the functional semantics of the code, and providing support for generating fine-grained ,practical code documents, finally, the target computing power scheduling engine code ,documents are generated based on the project tree and semantic information, and the ,code of the scheduling engine to be tested is organized into ,documents that are easy for engineers to read, so that engineers can quickly locate and fix problems.

[0111] Reference Figure 7 , Figure 7 It is a flowchart of a specific method for statically detecting multiple template description information in a target text template through a preset parser provided in an embodiment of the present application. The method includes but is not limited to steps S701 to S704.

[0112] Step S701: Perform mandatory item detection on multiple template description information in the target text template through a preset parser to obtain a first detection result.

[0113] Step S702: Perform key-value conflict detection on multiple template description information in the target text template through a preset parser to obtain a second detection result.

[0114] Step S703: Perform an accessibility test on multiple template description information in the target text template through a preset parser to obtain a third test result.

[0115] Step S704, obtaining a static detection result according to the first detection result, the second detection result and the third detection result.

[0116] In steps S701 to S704 of some embodiments, in the process of statically detecting multiple template description information in the target text template by a preset parser, the embodiments of the present application detect the required items of multiple template description information in the target text template by a preset parser. Specifically, a template scan is performed on all template description information in the target text template, all required tags are identified, and it is checked whether all fields marked as required in the template description information have been fully filled in, that is, whether the field corresponding to the required tag is empty is determined, and a first detection result is obtained, thereby realizing the detection of the required fields in the target text template, and avoiding errors caused by lack of key information in subsequent processing. The invention relates to a method for preventing errors or incomplete processing flows; performing key value conflict detection on multiple template description information in the target text template through a preset parser, specifically, ensuring that each key value of the template description information in the target text template is unique and that the value corresponding to each key has no logical conflict, thereby realizing the detection of the key values ​​in the target text template and obtaining a second detection result, thereby avoiding duplication or conflict of key values ​​in the target text template; performing accessibility testing on multiple template description information in the target text template through a preset parser, specifically, performing accessibility testing on all URL addresses contained in the template, and the embodiment of the present application sends HTTP (Hyper Text Transfer Protocol) requests to these URLs and checks the response status code to confirm whether they can be successfully accessed, thereby obtaining a third detection result, thereby ensuring the validity of each link in the template and avoiding user confusion or errors caused by link failure. Finally, a static detection result is obtained according to the first detection result, the second detection result and the third detection result, thereby improving the quality and reliability of the target text template, reducing runtime errors, and improving the overall performance of the system and user satisfaction.

[0117] Reference Figure 8 , Figure 8 It is a flowchart of a specific method for evaluating and verifying a preset programming large model based on fine-tuning model weights and a preset test file set provided in an embodiment of the present application. The method includes but is not limited to steps S801 to S802.

[0118] Step S801, input a preset test file set into a preset programming large model, so that the preset programming large model evaluates and verifies the test file set based on the fine-tuning model weights to obtain an evaluation result.

[0119] Step S802: When the evaluation result meets the preset evaluation condition, the preset programming large model is determined as the target programming large model.

[0120] In steps S801 to S802 of some embodiments, after the model fine-tuning is completed, the embodiments of the present application use an independent test data set to evaluate the performance of the model. Specifically, the preset test file set is input into the preset programming big model, so that the preset programming big model evaluates and verifies the test file set based on the fine-tuned model weights, checks the performance of the model on the test file set, and obtains an evaluation result. When the evaluation result meets the preset evaluation conditions, it means that the fine-tuned preset programming big model achieves the preset effect. The preset programming big model can be directly determined as the target programming big model to ensure that the performance of the big model on specific tasks is effectively improved and the fine-tuning achieves the expected effect.

[0121] It should be noted that the preset evaluation conditions in the embodiments of the present application can be set according to the needs of the user, and the embodiments of the present application do not impose specific limitations.

[0122] In order to more clearly illustrate the above-mentioned method for generating a computing power scheduling engine based on a large programming model, a specific example is given below.

[0123] Example 1:

[0124] Example 1 uses the programming big model service to make the scheduling engine development simpler and more efficient. The following is a detailed description of the computing power scheduling engine generation method based on the programming big model in this example.

[0125] Reference Fig. 9 , Fig. 9 It is a structural diagram of a large programming model provided in an example of this application.

[0126] In some embodiments, the programming big model is divided into three layers: big model base, big model fine-tuning, and big model service. The programming big model base includes commonly used CodeQwen, CodeFuse, and MiLM (Mi Language Model l). The programming big model generally includes an embedding layer, a multi-layer decoder, and a linear output layer. Each layer of the decoder includes a self-attention module and a multi-layer perceptron module, as well as layer normalization of the input and output.

[0127] Among them, programming large model fine-tuning is divided into data sets, fine-tuning techniques and monitoring. The data sets include annotated Octopus scheduling, ModelArts and SLURM code implementation. Specifically, fine-tuning techniques include data set preprocessing, parameter fine-tuning and inference testing.

[0128] In some embodiments, during the fine-tuning of the programming large model, first, based on the data set preprocessing, the data set preprocessing function and the single-row data preprocessing function are defined to map the original data set into data content suitable for large model fine-tuning, and then the parameter fine-tuning creates the fine-tuning parameters, sets the weight file directory after fine-tuning, and sets the SwanLab callback function to monitor the fine-tuning process. After the inference test fine-tuning is completed, the model is evaluated and verified using the test data to ensure that the fine-tuning achieves the expected effect.

[0129] Understandably, monitoring includes tools such as SwanLab, TensorBoard, and Wandb.

[0130] In some embodiments, the programming big model service includes code generation, code optimization, code testing, and code document generation, etc. Among them, automatic code generation means that the programming big model can automatically generate code according to demand, helping developers save time and energy; code optimization means that the programming big model can optimize the existing code to improve the readability and maintainability of the code; code testing means that the programming big model can automatically generate test cases to help developers find defects in the code; code document generation means that the programming big model can automatically generate code documents to help developers understand the results and functions of the code.

[0131] Example 2:

[0132] Reference Fig.10 , Fig.10 This is a schematic diagram of using a large programming model to achieve computing power scheduling fusion provided in an example of this application.

[0133] This example fine-tunes the large model base, uses the annotated Octopus scheduling, ModelArts, and SLURM code data sets to train the base large model, and obtains a large model dedicated to the scheduling system fusion. Then, by asking users to fill in natural language templates, the fine-tuned large model completes the template parsing and code generation, and then inserts the scheduling code generated by the large model into the original scheduling engine through the packager. After the code is merged, the testing function of the development tool can quickly implement local testing, cluster testing, and deployment testing of the scheduling engine to ensure the quality and efficiency of the development code. At the same time, the documentation function of the development tool can organize the scheduling engine code into documents that are easy for engineers to read, so that engineers can quickly locate and fix problems.

[0134] Each step is described in detail below.

[0135] Reference Figures 11a to 11d , Fig.11a is a schematic diagram of an example of an authentication description provided in an example of this application; Fig.11b is a schematic diagram of a resource description example provided in an example of this application; Fig.11cis a schematic diagram of an image description example provided in an example of this application; Fig.11d It is a schematic diagram of a task description example provided in an example of this application.

[0136] In some embodiments, the description of the fusion scheduling system includes authentication description, resource description, image description, and task description. The description is filled in according to the JSON format. Fig.11a , the authentication description provides the Fusion Scheduling System account and password information. Fig.11b , Resource Description provides descriptions of all resources in the fusion scheduling system, including name, creation time, computing power type, etc. Fig.11c , Image description provides the image description of the fusion scheduling system, including name, creation time, computing power type, etc. Fig.11d ,Task description provides the task description of the fusion scheduling system, including name, resource name, image name, center name, etc.

[0137] Reference Figures 12a to 12d , Fig.12a It is a comparison chart of parsing the authentication description provided by an example of this application; Figure 12b is a comparison chart of parsing resource descriptions provided in an example of this application; Fig.12c This is a comparison diagram of parsing the image description provided by an example of this application; Fig.12d This is a comparison chart of parsing a task description provided in an example of this application.

[0138] After obtaining different descriptions of the fusion scheduling system, the parser in the embodiment of the present application is responsible for translating the natural language template into a JSON file, wherein the natural language template includes four parts: authentication card, resource description card, image description card, and task description card. The parsed JSON file corresponds to each of them and also includes four parts. Figures 12a to 12d The translated format file mainly includes four key values, namely authentication, resources, images and tasks, where the authentication value is a simple dictionary, and the values ​​of resources, images and tasks are lists composed of multiple elements. That is, the JSON file after the requirements are parsed corresponds to the four description cards one by one, and also includes four parts.

[0139] It mainly includes four main key values, namely authentication, resources, images and tasks. The authentication value is a simple dictionary, and the values ​​of resources, images and tasks are lists composed of multiple elements.

[0140] The parser is also responsible for static detection of the natural language template format, including mandatory field detection, key-value conflict detection, and URL address accessibility testing.

[0141] In some embodiments, the scheduling system programming model API (Application Programming Interface) is integrated in the development environment, and the JSON file output in the previous step is input into the programming model to automatically generate the final scheduling system code. Among them, the generated code is mainly concentrated in the scheduling system client, and a small amount of code is distributed in the service adapter module and the launcher.

[0142] After the scheduling system code is generated, the code packaging module is responsible for embedding the generated scheduling system code patch into the scheduling engine. In software development, a code patch usually refers to a modification to an existing code base to fix errors, update functions, improve performance, or adapt to new development standards. Patches can be small changes, such as fixing a typo, or large changes, such as refactoring an entire system module.

[0143] Use a version control system to manage patches to ensure code traceability and rollback. Also record patch details, including the reason for the change, the scope of impact, and test results. Before applying the patch, conduct a code review to ensure the quality and security of the patch. Code patches are an important part of the software development process, and they help keep the software healthy and vibrant.

[0144] After the integration is completed, you can test it with one click. The test includes local test, cluster test and deployment test. This allows the computing power scheduling engine to truly achieve automated integration.

[0145] In the software development life cycle, testing is a key link to ensure software quality and performance. Local testing, cluster testing, and deployment testing are three different stages of testing, each with its own specific purpose and focus.

[0146] Local testing is usually performed by developers in the development environment to verify the correctness of the smallest testable unit of code. Use unit testing frameworks (such as JUnit, pytest, NUnit) to write test cases. Run automated tests to ensure that code changes do not break existing functionality. Ensure that a single function or method works as expected. Quickly discover and fix errors in the code. Thus improving code quality.

[0147] Cluster testing occurs when multiple components or services are integrated together to verify their interaction and collaboration as a whole. Ensure that different services or modules can integrate and communicate correctly. Detect compatibility issues between interfaces and services to verify the overall functionality and performance of the system. Deploy services in a simulated production environment. Use integration testing tools (such as Postman, SoapUI) for interface testing. Perform end-to-end testing to simulate user operations and business processes. Cluster testing may require a more complex test environment and configuration. The isolation and security of test data need to be ensured.

[0148] Deployment testing is the final stage of software testing before it is released to the production environment. It aims to ensure the stability and reliability of the software in the production environment. Verify the performance of the software in the production environment. Deploy the software in the pre-production environment. Perform performance testing, load testing, and stress testing, and conduct security testing to ensure the security of the software.

[0149] These three testing phases are complementary, and each phase focuses on different aspects of software quality. Local testing focuses on the correctness of code units, cluster testing focuses on the integration between components, and deployment testing focuses on the performance of the software in the production environment. An effective testing strategy should include all of these phases to ensure software quality and user satisfaction.

[0150] Through local testing, cluster testing, and deployment testing, the computing power scheduling engine can achieve high-quality integration.

[0151] After code optimization is completed, let the programming big model automatically generate code documentation. The big model uses global structure parsing technology, through deep recursion and abstract syntax tree analysis, to accurately parse the classes and functions in each file and build a project tree. It also parses the bidirectional reference relationship between code objects, providing a global contextual perspective for the big model to assist in identifying the functional semantics of the code and provide support for generating fine-grained practical code documentation.

[0152] It should be noted that during the document generation phase, the large model generates fine-grained code documentation, which includes the function description, parameter description, code description, precautions and usage examples of the code object.

[0153] In some embodiments, the embodiments of the present application can simplify the interface fusion process of different sub-scheduling systems. By taking advantage of the large programming model to automatically generate code and automatically generate documents for learning demand templates, development engineers can complete the rapid fusion of the scheduling system without having to deeply understand the distribution and functions of all sub-modules of the scheduling engine, making the development process more efficient, the test coverage more comprehensive, and improving the maintainability of the system. This method significantly reduces the development and testing workload, and provides an efficient solution for the computing power scheduling engine in terms of multi-system integration.

[0154] Reference Fig.13 , taking the control processor 1001 and memory 1002 in the electronic device 1000 as an example that can be connected via a bus. The memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk memory, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the control processor 1001, and these remote memories may be connected to the electronic device 1000 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0155] Those skilled in the art will understand that Fig.13 The device structure shown in the figure does not constitute a limitation on the electronic device 1000, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0156] The embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by one or more control processors, for example, Fig.13 One of the control processors 1001 executes, which can enable the above-mentioned one or more control processors to execute the computing power scheduling engine generation method based on the programming big model in the above-mentioned method embodiment.

[0157] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A method for generating a computing power scheduling engine based on a programming large model, characterized in that: include: Obtain a target text template, where the target text template is filled with a plurality of template description information of the computing power scheduling engine relative to the computing power scheduling system; According to a target data exchange format, converting the plurality of template description information in the target text template into a target format file; Inputting the target format file into the target programming big model to obtain the scheduling system code for the computing power scheduling system; wherein the target programming big model is obtained by fine-tuning the preset programming big model based on the sample annotated file set; Obtaining a preset scheduling engine code, and fusing the scheduling system code with the preset scheduling engine code to obtain a scheduling engine code to be tested; Testing the scheduling engine code to be tested to obtain a test result; When the test result is that the test passes, a target computing power scheduling engine code document is generated for the scheduling engine code to be tested that passes the test.

2. The method for generating a computing power scheduling engine based on a programming big model according to claim 1, characterized in that: The target programming model is obtained by the following steps: Performing data mapping on the sample annotation file set through a preset preprocessing function to obtain a fine-tuning data set; Creating fine-tuning parameters, fine-tuning the preset programming model based on the fine-tuning parameters and the fine-tuning data set, and monitoring the fine-tuning process through a preset callback function to obtain a fine-tuning model weight; The preset programming large model is evaluated and verified according to the fine-tuning model weights and the preset test file set to obtain the target programming large model.

3. The method for generating a computing power scheduling engine based on a programming big model according to claim 1, characterized in that: The converting the plurality of template description information in the target text template into a target format file according to the target data exchange format includes: Performing static detection on multiple template description information in the target text template through a preset parser to obtain a static detection result; When the static detection result is that the test is passed, the template description information that has passed the static detection is translated into the target data exchange format through the parser to obtain a target format file.

4. The method for generating a computing power scheduling engine based on a programming big model according to claim 1, characterized in that: The step of testing the scheduling engine code to be tested and obtaining a test result includes: Performing a local test on the scheduling engine code to be tested to obtain a first test result; Performing a cluster test on the scheduling engine code to be tested to obtain a second test result; Performing a deployment test on the scheduling engine code to be tested to obtain a third test result; A test result is obtained according to the first test result, the second test result and the third test result.

5. The method for generating a computing power scheduling engine based on a programming big model according to claim 1, characterized in that: After testing the scheduling engine code to be tested and obtaining the test result, the method further includes: When the test result is that the test fails, optimizing the scheduling engine code to be tested that fails the test to obtain a target scheduling engine code; The target scheduling engine code is tested to obtain a target test result.

6. The method for generating a computing power scheduling engine based on a programming big model according to claim 1, characterized in that: The step of generating a target computing power scheduling engine code document for the scheduling engine code to be tested that has passed the test includes: Performing global structural analysis on the tested scheduling engine code through the preset programming macro model to obtain an element list; Performing file analysis on the element list based on a preset deep recursive algorithm and an abstract syntax tree analysis method to obtain an abstract syntax tree; Constructing a project tree based on the abstract syntax tree, and performing bidirectional reference relationship analysis on the project tree and the element list to obtain semantic information; Generate a target computing power scheduling engine code document based on the project tree and the semantic information.

7. The method for generating a computing power scheduling engine based on a programming big model according to claim 1, characterized in that: The static detection of the multiple template description information in the target text template by a preset parser to obtain a static detection result includes: Performing mandatory item detection on multiple template description information in the target text template through a preset parser to obtain a first detection result; Performing key-value conflict detection on multiple template description information in the target text template through a preset parser to obtain a second detection result; Performing an accessibility test on multiple template description information in the target text template through a preset parser to obtain a third test result; A static detection result is obtained according to the first detection result, the second detection result and the third detection result.

8. The method for generating a computing power scheduling engine based on a programming big model according to claim 2, characterized in that: The step of evaluating and verifying the preset programming large model according to the fine-tuning model weight and the preset test file set to obtain the target programming large model includes: Inputting a preset test file set into the preset programming big model, so that the preset programming big model evaluates and verifies the test file set based on the fine-tuning model weights to obtain an evaluation result; When the evaluation result meets the preset evaluation condition, the preset programming big model is determined as the target programming big model.

9. An electronic device, characterized in that: It includes at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the computing power scheduling engine generation method based on the programming big model as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for generating a computing power scheduling engine based on a programming big model as described in any one of claims 1 to 8.

Images