Automatic homework arrangement method and device, equipment and storage medium
By parsing development requirements to generate search terms, retrieving target knowledge bases, and building job workflows, the problem of low efficiency in big data job orchestration is solved, and automated script generation and dependency configuration are achieved, thus improving development efficiency.
Patent Information
- Application Number
- CN202511664672.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-13
- Publication Date
- 2026-02-13
AI Technical Summary
In the existing technology, during the big data job orchestration process, developers need to manually navigate through processes and manually refresh data, resulting in low development efficiency and automation levels, and long script writing time.
By parsing development requirements to generate search terms, retrieving results from the target knowledge base, determining prompt word templates, generating target scripts and determining configuration dependencies, and building a job workflow, the system can automatically generate scripts and configuration dependencies for each process node.
It improves the efficiency and automation level of job scheduling for big data demand development, reduces the time developers spend manually writing scripts, and increases development efficiency and automation.
Smart Images
Figure CN121525657A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of big data, and particularly relates to a job automatic arrangement method and device, equipment and a storage medium. BACKGROUND
[0002] In the field of big data demand development, job arrangement needs to be performed for big data demand development to generate a corresponding job workflow, thereby meeting the big data demand development.
[0003] At present, the job arrangement for big data is performed in the following manner: a developer logs in a big data development platform to perform processes such as new table creation, new development script creation, dependency configuration, job case writing, and git submission, to form a corresponding job workflow, thereby realizing big data demand development.
[0004] However, in the process of generating the job workflow, on the one hand, the developer needs to manually jump to the next process after the previous process development is completed, and needs to manually refresh to obtain data (such as obtaining a corresponding git branch directory, and being unable to automatically obtain table information) in each process; on the other hand, the script in the job workflow needs to be manually written by the developer, which consumes a large amount of development time; in summary, the development efficiency and automation level of the job arrangement for big data demand development are both low. SUMMARY
[0005] In order to improve the development efficiency and automation level of the job arrangement for big data demand development, the present application provides a job automatic arrangement method, device, equipment and storage medium.
[0006] In a first aspect, the present application provides a job automatic arrangement method, comprising:
[0007] analyzing a development demand to obtain a to-be-retrieved term, and retrieving a target knowledge base based on the to-be-retrieved term to obtain a retrieval result;
[0008] determining a prompt word template matched with the retrieval result to obtain a target prompt word template;
[0009] generating a target script based on the retrieval result and the target prompt word template, and determining a configuration dependency based on the target script;
[0010] constructing a job workflow based on the target script and the configuration dependency.
[0011] By the above implementation, the development requirement of the developer for developing the workflow is converted into a search result, and then a prompt word is generated based on the search result, and the target script and the configuration dependency are generated through the prompt word. In this way, the scripts and configuration dependencies of each process node of the workflow can be automatically generated according to the development requirement, and the developer can avoid spending a lot of time writing scripts, thereby facilitating the effective improvement of the development efficiency and the automation level of the job scheduling development for the big data requirement development.
[0012] Preferably, the development requirement is analyzed to obtain a to-be-searched term, and the method comprises the following steps.
[0013] The development requirement is analyzed to obtain a professional term.
[0014] The professional term is replaced by a synonym to obtain a similar term.
[0015] The to-be-searched term is obtained based on the professional term and the similar term.
[0016] By the above implementation, the professional term and the similar term with similar semantics are jointly used as the to-be-searched term, which can improve the comprehensiveness and accuracy of the results searched based on the to-be-searched term.
[0017] Preferably, the prompt word template matched with the search result is determined to obtain a target prompt word template, and the method comprises the following steps.
[0018] The keywords in each to-be-selected prompt word template are extracted.
[0019] The matching degrees between the search result and the keywords corresponding to each to-be-selected prompt word template are calculated to obtain a matching degree set.
[0020] The target prompt word template is determined based on the matching degree set.
[0021] By the above implementation, the to-be-selected prompt word template corresponding to the maximum matching degree in the matching degree set can be used as the target prompt word template matched with the search result, thereby facilitating the improvement of the accuracy of the determined target prompt word template.
[0022] Preferably, the target script is generated based on the search result and the target prompt word template, and the method comprises the following steps.
[0023] The prompt word is obtained based on the search result and the target prompt word template.
[0024] The target script is obtained by processing the prompt word based on a preset language model.
[0025] By the above implementation, the target script can be obtained by processing the prompt word based on the language model, thereby effectively improving the efficiency and automation level of script generation.
[0026] Preferably, the preset language model is used to process the prompt word to obtain a target script, including:
[0027] The preset language model is used to process the prompt word to obtain an initial script;
[0028] The modified prompt word and the language model are used to process the initial script to obtain a target script.
[0029] Through the above implementation, the initial script can be further modified and improved by modifying the prompt word and the language model, so as to improve the accuracy of the generated target script.
[0030] Preferably, the configuration dependency is determined based on the target script, including:
[0031] The target script is subjected to static code analysis to determine the configuration dependency;
[0032] Or, the target script is subjected to running detection to determine the configuration dependency.
[0033] Through the above implementation, the configuration dependency of the target script is determined, which facilitates subsequent determination of a workflow based on the target script and the configuration dependency.
[0034] Preferably, the application provides a job automatic arrangement method, further including:
[0035] Based on the job workflow, workflow task information is determined, and the workflow task information includes: a number of workflow tasks, a task running time length;
[0036] Based on the workflow task information, resource estimated consumption is calculated.
[0037] Through the above implementation, the workflow resource estimated consumption is calculated in advance, which facilitates subsequent allocation of appropriate resources to the workflow during the workflow running process, so as to improve the stability of the workflow running.
[0038] In a second aspect, the application provides a job automatic arrangement device, including:
[0039] A knowledge retrieval module is configured to analyze a development demand to obtain a to-be-retrieved term, and retrieve a target knowledge base based on the to-be-retrieved term to obtain a retrieval result;
[0040] A template generation module is configured to determine a prompt word template matched with the retrieval result to obtain a target prompt word template;
[0041] A script and dependency determination module is configured to generate a target script based on the retrieval result and the target prompt word template, and determine a configuration dependency based on the target script;
[0042] a workflow generation module, configured to construct a job workflow based on the target script and the configuration dependency.
[0043] By the above implementation, the development requirement of the developer for developing the workflow is converted into the search result, and then the prompt word is generated based on the search result, and the target script and the configuration dependency are generated through the prompt word. In this way, the scripts of the nodes of the workflow and the configuration dependency can be automatically generated according to the development requirement, and the developer is also avoided from spending a lot of time in writing the scripts, so as to effectively improve the development efficiency and the automation level of the job arrangement for the development of the big data requirement.
[0044] In a third aspect, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0045] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps of the above method.
[0046] In a fifth aspect, a computer program product is also provided. The computer program product includes a computer program. The computer program is executed by a processor to implement the steps of any of the above method embodiments.
[0047] The above job automatic arrangement method, device, equipment and storage medium. The development requirement is analyzed to obtain a to-be-searched term, the target knowledge base is searched based on the to-be-searched term to obtain a search result, a target prompt word template is obtained by determining a prompt word template matched with the search result, a target script is generated based on the search result and the target prompt word template, a configuration dependency is determined based on the target script, and a job workflow is constructed based on the target script and the configuration dependency. By the above implementation, the development requirement of the developer for developing the workflow is converted into the search result, and then the prompt word is generated based on the search result, and the target script and the configuration dependency are generated through the prompt word. In this way, the scripts of the nodes of the workflow and the configuration dependency can be automatically generated according to the development requirement, and the developer is also avoided from spending a lot of time in writing the scripts, so as to effectively improve the development efficiency and the automation level of the job arrangement for the development of the big data requirement.
[0048] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS
[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope. Other related drawings can also be obtained by those of ordinary skill in the art without creative labor on the basis of these drawings.
[0050] Figure 1 A flow chart of a job automatic scheduling method provided in an embodiment of the present application;
[0051] Figure 2 A structural schematic diagram of a job automatic scheduling device provided in an embodiment of the present application;
[0052] Figure 3 A structural schematic diagram of a computer device provided in an embodiment of the present application;
[0053] Figure 4 An internal structural diagram of a computer readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and not to limit the present disclosure.
[0055] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or apparatus that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatuses.
[0056] In this paper, the term "and / or" is only a description of the relationship between the associated objects, which means that there can be three relationships. For example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents a "or" relationship between the front and rear associated objects.
[0057] Embodiment one
[0058] Figure 1A work automatic scheduling method flowchart is provided for Embodiment I of the present application, reference is made to Figure 1 The method can be executed by a device for executing the method, which can be realized by software and / or hardware. The method comprises the following steps:
[0059] S110, the development requirement is analyzed to obtain a to-be-retrieved term, and a target knowledge base is retrieved based on the to-be-retrieved term to obtain a retrieval result.
[0060] The development requirement is a requirement text in natural language form proposed by a developer for a development workflow. A script corresponding to the development requirement needs to be generated using the large model, and therefore the development requirement needs to be converted from natural language to data understandable by the large model.
[0061] Specifically, the development requirement is analyzed using a preset semantic model, so as to convert the development requirement into a plurality of terms, and the terms are recorded as to-be-retrieved terms. It should be noted that the to-be-retrieved terms cannot make the large model generate a corresponding script, and the to-be-retrieved terms also need to be used to perform corresponding retrieval in a knowledge base, so as to generate input information required by the large model for generating a script. The knowledge base is a knowledge base for real-time knowledge updating, so as to ensure the real-time and accuracy of the knowledge data in the knowledge base, and the knowledge base is recorded as a target knowledge base. It should be noted that the target knowledge base includes a large amount of data related manual documents. The result obtained by the to-be-retrieved terms performing corresponding retrieval in the knowledge base is the retrieval result.
[0062] S120, a prompt word template matching the retrieval result is determined to obtain a target prompt word template.
[0063] It should be noted that the present embodiment intends to input a prompt word into the large model, so as to make the large model generate a corresponding script. Therefore, the prompt word needs to be generated.
[0064] In order to generate the prompt word, the present embodiment presets a prompt word template matching the retrieval result, and the retrieval result can be used as filling content filled into the prompt word template. The prompt word template matching the retrieval result is recorded as a target prompt word template.
[0065] S130, a target script is generated based on the retrieval result and the target prompt word template, and configuration dependency is determined based on the target script.
[0066] The prompt word can be used to guide the large model to generate scripts of each process node of the workflow, and the generated script is recorded as a target script; in order to run the target script subsequently, configuration dependencies of each target script are also determined, wherein the configuration dependencies are external configurations, resources or environmental conditions necessary for the target script when running; in this embodiment, the configuration dependencies include environment variables, configuration files, external services and network endpoints, file system paths and the like; the configuration dependencies can be obtained by analyzing the target script.
[0067] S140, constructing a job workflow based on the target script and the configuration dependencies.
[0068] The job workflow is a complete DAG definition file that can be scheduled and executed by a workflow engine, and clearly defines the task execution order, dependencies and trigger conditions; the job workflow can be obtained by parsing the target script and the corresponding configuration dependencies.
[0069] It should be noted that, in this embodiment, the development requirement is analyzed to obtain a to-be-retrieved term, the target knowledge base is retrieved based on the to-be-retrieved term to obtain a retrieval result, a prompt word template that matches the retrieval result is determined to obtain a target prompt word template, a target script is generated based on the retrieval result and the target prompt word template, configuration dependencies are determined based on the target script, and a job workflow is constructed based on the target script and the configuration dependencies. Through the above implementation, the development requirement of the developer for developing the workflow is converted into the retrieval result, and then the prompt word is generated based on the retrieval result, the target script and the configuration dependencies are generated through the prompt word, so that the scripts and configuration dependencies of each process node of the workflow can be automatically generated according to the development requirement, and the developer is also saved from spending a lot of time on writing scripts, thereby effectively improving the development efficiency and the automation level of the job arrangement for the development of the big data requirement.
[0070] Embodiment Two
[0071] The method provided in Embodiment Two of the present application optimizes the "analyzing the development requirement to obtain a to-be-retrieved term" in Embodiment One; it should be noted that the parts not described in this embodiment can be referred to the descriptions of other embodiments, and the method comprises:
[0072] S211, analyzing the development requirement to obtain a professional term.
[0073] In this embodiment, a semantic model is preset, which is used to analyze the input development requirement to obtain the professional term in the development requirement.
[0074] S212, performing synonym replacement on the professional term to obtain a similar term.
[0075] It should be noted that the professional term is used for subsequent retrieval in the target knowledge base to obtain the corresponding retrieval result; in order to improve the comprehensiveness and accuracy of the retrieval result, the embodiment also determines the synonyms of the professional term, and the professional term and its synonyms are used for subsequent retrieval.
[0076] Specifically, by synonym replacement of the professional term, synonyms of the professional term can be obtained, and the synonyms of the professional term are recorded as similar terms.
[0077] Through the above implementation, the professional term and the similar term similar in semantics are collectively used as the to-be-retrieved term, which can improve the comprehensiveness and accuracy of the result retrieved by the to-be-retrieved term subsequently.
[0078] S213, obtaining a to-be-retrieved term based on the professional term and the similar term.
[0079] The to-be-retrieved term includes the professional term and the similar term.
[0080] S214, retrieving a target knowledge base based on the to-be-retrieved term to obtain a retrieval result.
[0081] S220, determining a prompt word template matched with the retrieval result to obtain a target prompt word template.
[0082] S230, generating a target script based on the retrieval result and the target prompt word template, and determining a configuration dependency based on the target script.
[0083] S240, constructing a job workflow based on the target script and the configuration dependency.
[0084] Embodiment three
[0085] The embodiment three of the application provides a job automatic arrangement method, which optimizes the "determining a prompt word template matched with the retrieval result to obtain a target prompt word template" in the embodiment one; it should be noted that the parts not described in the embodiment can be referred to the description of other embodiments, and the method comprises:
[0086] S310, obtaining a to-be-retrieved term by analyzing a development requirement, and retrieving a target knowledge base based on the to-be-retrieved term to obtain a retrieval result.
[0087] S321, extracting a keyword in each to-be-selected prompt word template.
[0088] It should be noted that the retrieval result is used for subsequent filling into a preset prompt word template matched with the retrieval result, and for this purpose, the preset prompt word template most matched with the retrieval result needs to be determined from a plurality of preset prompt word templates.
[0089] Specifically, taking one of the prompt word templates as an example, the keywords in each prompt word template are extracted first, and exemplary keywords are, for example, Python, code, function, and the like.
[0090] S322, calculate the matching degree between the search result and the keywords corresponding to each of the candidate prompt word templates, and obtain a matching degree set.
[0091] It should be noted that the matching degree between the search result and the keywords corresponding to the prompt word template it matches is the highest; wherein, and the preset prompt word templates are recorded as candidate prompt word templates; the matching degree between the search result and the keywords corresponding to the prompt word template can be the cosine similarity between the first vector corresponding to the search result and the second vector corresponding to the keywords.
[0092] Through the above implementation, the candidate prompt word template corresponding to the maximum matching degree in the matching degree set can be used as the target prompt word template matched by the search result, so as to improve the accuracy of the determined target prompt word template.
[0093] S323, determine a target prompt word template based on the matching degree set.
[0094] The matching degree set includes the set of matching degrees obtained after the search result and each candidate prompt word template are matched; the target prompt word template is the candidate prompt word template corresponding to the maximum matching degree in the matching degree set.
[0095] S330, generate a target script based on the search result and the target prompt word template, and determine a configuration dependency based on the target script.
[0096] S340, construct a job workflow based on the target script and the configuration dependency.
[0097] Embodiment Four
[0098] The job automatic arrangement method provided in Embodiment Four optimizes the "generating a target script based on the search result and the target prompt word template" in Embodiment One; it should be noted that the parts not described in this embodiment can be referred to the descriptions of other embodiments, and the method comprises:
[0099] S410, analyze the development requirement to obtain a to-be-searched term, and search a target knowledge base based on the to-be-searched term to obtain a search result.
[0100] S420, determine a prompt word template matched with the search result to obtain a target prompt word template.
[0101] S431. Based on the search results and the target prompt word template, prompt words are obtained.
[0102] The search results are used to populate the target prompt template to form corresponding prompts; for example, the target prompt template is a structured prompt template, as shown below:
[0103] [Role] You are an expert with 10 years of big data experience.
[0104] [Task] Generate code for {language} based on the following input:
[0105] {Input Requirements}
[0106] [Constraints] Must use {technology stack} and follow {code style}
[0107] [Example] Reference case: {Similar case code snippet}.
[0108] Among them, language, input requirements, technology stack, code style, and similar case code snippets can be obtained from the search results.
[0109] S432. Process the prompt words based on a preset language model to obtain the target script.
[0110] In this embodiment, a language model is preset to process the prompt word. By processing the prompt word, the language model can sequentially generate scripts corresponding to each process node of the workflow, and record the script as the target script.
[0111] By implementing the above methods, the target script is obtained by processing the prompt words through a language model, which can effectively improve the efficiency and automation level of script generation.
[0112] S433. Determine configuration dependencies based on the target script.
[0113] S440. Based on the target script and the configuration dependencies, construct the job workflow.
[0114] Example 5
[0115] This application provides a method for automatically scheduling jobs in Embodiment 5, which optimizes the "generating a target script based on the search results and the target prompt word template" in Embodiment 4. It should be noted that for parts not detailed in this embodiment, please refer to the descriptions in other embodiments. The method includes:
[0116] S510. The development requirements are analyzed to obtain the terms to be retrieved, and the target knowledge base is searched based on the terms to be retrieved to obtain the search results.
[0117] S520, determine a prompt word template matched with the search result to obtain a target prompt word template.
[0118] S531, obtain a prompt word based on the search result and the target prompt word template.
[0119] S532A, process the prompt word based on a preset language model to obtain an initial script.
[0120] Wherein, and the script obtained by processing the prompt word based on the language model is recorded as the initial script.
[0121] It should be noted that the initial script is directly generated by the language model, and there may be some places that need to be modified; if not modified, the script may be incorrect.
[0122] S532B, process the initial script based on the modified prompt word and the language model to obtain a target script.
[0123] It should be noted that, in order to prevent script errors, the embodiment further checks and modifies the initial script through the language model after the language model generates the initial script. In order to control the language model to continue to check and modify the initial script, the embodiment is preset with a modified prompt word. For example, the modified prompt word is "please check and modify the following script comprehensively, and give a reply (code analysis, problem list, modified code, modification description) according to the steps", and the script output by the language model after completing the modification of the initial script is recorded as the target script.
[0124] Through the above implementation, the initial script can be further modified and improved through the modified prompt word and the language model processing, so as to improve the accuracy of the generated target script.
[0125] S533, determine configuration dependency based on the target script.
[0126] S540, build a job workflow based on the target script and the configuration dependency.
[0127] Embodiment six
[0128] The job automatic arrangement method provided in embodiment six optimizes "determining configuration dependency based on the target script" in embodiment one; it should be noted that the parts not described in this embodiment can be referred to the description of other embodiments. The method comprises:
[0129] S610, analyze the development requirement to obtain a to-be-searched term, and search a target knowledge base based on the to-be-searched term to obtain a search result.
[0130] S620, determine a prompt word template matched with the search result to obtain a target prompt word template.
[0131] S631, generating a target script based on the search result and the target prompt word template.
[0132] S632, performing static code analysis on the target script to determine configuration dependency.
[0133] The configuration dependency is obtained through the target script. The embodiment provides two ways of obtaining the configuration dependency through the target script. One way is to obtain the configuration dependency by performing static code analysis on the target script. The static code analysis is to directly analyze the target script to determine the configuration dependency, such as library dependency and network endpoint dependency directly contained in the target script.
[0134] S633, or, performing running detection on the target script to determine the configuration dependency.
[0135] The embodiment also provides a way of determining the configuration dependency of the target script, that is, determining the configuration dependency by performing running detection on the target script.
[0136] Specifically, a clean container or virtual machine is configured first, and then the target script is run in the container or virtual machine. The clean environment almost lacks all dependencies, and the script will quickly fail due to missing items. By observing the error information, all hidden configuration dependencies can be found one by one.
[0137] Through the above implementation, the configuration dependency of the target script is determined, which facilitates subsequent determination of a workflow based on the target script and the configuration dependency.
[0138] S640, constructing a job workflow based on the target script and the configuration dependency.
[0139] Embodiment seven
[0140] The embodiment seven of the present application provides a job automatic arrangement method, which supplements the method shown in embodiment one. It should be noted that the parts not described in detail in the present embodiment can be referred to the descriptions of other embodiments. The method comprises:
[0141] S710, analyzing a development requirement to obtain a to-be-searched term, and searching a target knowledge base based on the to-be-searched term to obtain a search result.
[0142] S720, determining a prompt word template matched with the search result to obtain a target prompt word template.
[0143] S730, generating a target script based on the search result and the target prompt word template, and determining configuration dependency based on the target script.
[0144] S740, constructing a job workflow based on the target script and the configuration dependency.
[0145] S750, determining workflow task information based on the job workflow, the workflow task information comprising: a number of workflow tasks, a task runtime length.
[0146] Wherein, the job workflow comprises a plurality of tasks, and the tasks consume resources (CPU resources, memory resources, network resources, etc.) when running. In order to ensure that appropriate resources can be allocated to each task, the resources required by the job workflow can be estimated in advance before the job workflow runs.
[0147] Specifically, after the job workflow is constructed, each task contained in the job workflow can be further determined, denoted as a workflow task, and the number of workflow tasks is denoted as the number of workflow tasks. The historical runtime length of the workflow task is denoted as the task runtime length.
[0148] S760, calculating resource estimation consumption based on the workflow task information.
[0149] It should be noted that the present embodiment has a calculation formula for calculating resource estimation consumption, as follows:
[0150] SQL estimated running waiting completion length = (number of workflow tasks + 1) x P95 task runtime length;
[0151] Resource estimation consumption = (SQL estimated running waiting completion length x average resource consumption) / SQL average completion time;
[0152] Wherein, P95 task runtime length is the 95th percentile of the task runtime length corresponding to the plurality of tasks in the workflow; average resource consumption and SQL average completion time are preset statistical values.
[0153] Through the above implementation, the workflow resource estimation consumption is calculated in advance, which facilitates subsequent allocation of appropriate resources to the workflow during workflow running, thereby improving the stability of workflow running.
[0154] It should be understood that although each step in the flowchart involved in each of the above-described embodiments is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowchart involved in each of the above-described embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least some of the other steps or steps or stages in other steps.
[0155] Embodiment Eight
[0156] Based on the same inventive concept, the present embodiment also provides a job automatic arrangement device for implementing the above-mentioned job automatic arrangement method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more job automatic arrangement device embodiments provided below can refer to the limitations of the job automatic arrangement method in the above text, which will not be repeated here.
[0157] In the present embodiment, as shown in Figure 2 a job automatic arrangement device is provided, comprising:
[0158] a knowledge retrieval module configured to analyze the development demand to obtain a to-be-retrieved term, and retrieve a target knowledge base based on the to-be-retrieved term to obtain a retrieval result;
[0159] a template generation module configured to determine a prompt word template matched with the retrieval result to obtain a target prompt word template;
[0160] a script and dependency determination module configured to generate a target script based on the retrieval result and the target prompt word template, and determine a configuration dependency based on the target script;
[0161] a workflow generation module configured to construct a job workflow based on the target script and the configuration dependency.
[0162] Each module in the above job automatic arrangement device can be realized by software, hardware, and a combination thereof, in whole or in part. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.
[0163] It should be noted that the embodiment obtains the to-be-retrieved terms by analyzing the development requirement, obtains the retrieval result based on the to-be-retrieved terms, determines the target prompt word template matched with the retrieval result, generates the target script based on the retrieval result and the target prompt word template, determines the configuration dependency based on the target script, and constructs the job workflow based on the target script and the configuration dependency. Through the above implementation, the development requirement of the developer in developing the workflow is converted into the retrieval result, and then the prompt word is generated based on the retrieval result, the target script and the configuration dependency are generated through the prompt word. In this way, the script and the configuration dependency of each process node of the workflow can be automatically generated according to the development requirement, and the developer can avoid spending a lot of time in writing scripts, thereby facilitating the effective improvement of the development efficiency and the automation level of the job arrangement for the development of the big data requirement.
[0164] In an optional embodiment, the obtaining the to-be-retrieved terms by analyzing the development requirement comprises:
[0165] analyzing the development requirement to obtain professional terms;
[0166] performing synonym replacement on the professional terms to obtain similar terms;
[0167] obtaining the to-be-retrieved terms based on the professional terms and the similar terms.
[0168] In an optional embodiment, the determining the target prompt word template matched with the retrieval result comprises:
[0169] extracting keywords in each candidate prompt word template;
[0170] calculating a matching degree between the retrieval result and the keywords corresponding to each candidate prompt word template to obtain a matching degree set;
[0171] determining the target prompt word template based on the matching degree set.
[0172] In an optional embodiment, the generating the target script based on the retrieval result and the target prompt word template comprises:
[0173] obtaining a prompt word based on the retrieval result and the target prompt word template;
[0174] processing the prompt word based on a preset language model to obtain the target script.
[0175] In an optional embodiment, the processing the prompt word based on the preset language model to obtain the target script comprises:
[0176] processing the prompt word based on a preset language model to obtain an initial script;
[0177] processing the initial script based on the modified prompt word and the language model to obtain a target script.
[0178] In an optional embodiment, the determining configuration dependency based on the target script comprises:
[0179] performing static code analysis on the target script to determine the configuration dependency;
[0180] or, performing runtime detection on the target script to determine the configuration dependency.
[0181] In an optional embodiment, the job automatic arrangement method further comprises:
[0182] determining workflow task information based on the job workflow, the workflow task information comprising: a number of workflow tasks, a task runtime length;
[0183] calculating resource estimated consumption based on the workflow task information.
[0184] Embodiment Nine
[0185] In this embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 3 The computer device comprises a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a job automatic arrangement method.
[0186] Those skilled in the art can understand that Figure 3 the structure shown in the figure is only a block diagram of part of the structure related to the present disclosure, and does not constitute a limitation on the computer device to which the present disclosure is applied. Specifically, the computer device can comprise more or fewer components than those shown in the figure, or some components can be combined, or have a different component arrangement.
[0187] Embodiment Ten
[0188] In this embodiment, a computer readable storage medium is provided, as shown in Figure 4 which stores a computer program. The computer program is executed by the processor to implement the steps in each method embodiment described above.
[0189] Embodiment eleven
[0190] In this embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps in the above method embodiments.
[0191] It should be noted that the collected information is information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards of the country and region, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user selection authorization or refusal.
[0192] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present disclosure can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present disclosure can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.
[0193] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, as long as the combinations of technical features do not have contradictions, they shall be considered within the scope of the present disclosure.
[0194] The above embodiments only express several implementation manners of the present disclosure, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the patent scope of the present disclosure. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these shall be within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the appended claims.
Claims
1. A method for automatically scheduling jobs, characterized in that, include: The development requirements are analyzed to obtain the terms to be retrieved, and the target knowledge base is searched based on the terms to be retrieved to obtain the search results; The target prompt word template is obtained by determining the prompt word template that matches the search results; A target script is generated based on the search results and the target prompt word template, and configuration dependencies are determined based on the target script. Based on the target script and the configuration dependencies, a job workflow is constructed.
2. The method according to claim 1, characterized in that, The process of parsing development requirements to obtain the terms to be retrieved includes: Analyze development requirements to obtain technical terminology; The technical terms were replaced with synonyms to obtain similar terms; Based on the aforementioned specialized terminology and similar terms, the term to be retrieved is obtained.
3. The method according to claim 1, characterized in that, The process of determining the prompt word template that matches the search results to obtain the target prompt word template includes: Extract keywords from each candidate suggestion template; Calculate the degree of matching between the search results and the keywords corresponding to each of the candidate suggestion word templates to obtain a set of matching degrees; Based on the matching degree set, the target prompt word template is determined.
4. The method according to claim 1, characterized in that, The step of generating a target script based on the search results and the target prompt word template includes: Based on the search results and the target prompt word template, prompt words are obtained; The prompt words are processed based on a preset language model to obtain the target script.
5. The method according to claim 4, characterized in that, The process of processing the prompt words based on a preset language model to obtain the target script includes: The prompt words are processed based on a preset language model to obtain an initial script; The initial script is processed based on the modified prompt words and the language model to obtain the target script.
6. The method according to claim 1, characterized in that, The process of determining configuration dependencies based on the target script includes: Perform static code analysis on the target script to determine configuration dependencies; Alternatively, the target script can be run to determine configuration dependencies.
7. The method according to claim 1, characterized in that, Also includes: Based on the job workflow, determine the workflow task information, which includes: the number of workflow tasks and the task runtime. Based on the workflow task information, the estimated resource consumption is calculated.
8. An automatic job scheduling device, characterized in that, The device includes: The knowledge retrieval module is used to parse the development requirements to obtain the terms to be retrieved, and to retrieve the target knowledge base based on the terms to be retrieved to obtain the retrieval results; The template generation module is used to determine the prompt word template that matches the search results to obtain the target prompt word template; The script and dependency determination module is used to generate a target script based on the search results and the target prompt word template, and to determine configuration dependencies based on the target script. The workflow generation module is used to construct a job workflow based on the target script and the configuration dependencies.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.