Flink task management method, electronic device, and computer-readable storage medium

Abstract

The present application discloses a Flink task management method, an electronic device, and a computer-readable storage medium, the method comprising: obtaining a configured task and determining the configuration information of the task; wherein the configuration information includes at least configuration items related to memory; based on the configuration items and the task memory model corresponding to the task, determining the memory to be verified of at least part of the memory area in the task memory model; verifying the memory to be verified and generating a verification log, and managing the task based on the verification log; wherein, when the memory to be verified of all the obtained memory areas passes the verification, the task can be submitted for operation. In the above manner, the present application can improve the memory verification feedback efficiency and the stability of task operation.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a Flink task management method, an electronic device, and a computer-readable storage medium. Background Art

[0002] As the demand for data real-time performance in data applications gradually increases, most real-time platforms are built based on Flink, thanks to Flink's maturity in the real-time field. However, in existing technologies, newly developed Flink tasks usually wait for error information during the running process after submission to determine the memory verification results, resulting in low memory verification feedback efficiency and affecting the stability of task operation when the memory configuration is unreasonable. In view of this, how to improve the memory verification feedback efficiency and the stability of task operation has become an urgent problem to be solved. Summary of the invention

[0003] The main technical problem solved by this application is to provide a Flink task management method, an electronic device and a computer-readable storage medium, which can improve the memory verification feedback efficiency and the stability of task operation.

[0004] To solve the above technical problems, the first aspect of the present application provides a Flink task management method, including: obtaining a configured task and determining the configuration information of the task; wherein the configuration information includes at least configuration items related to memory; based on the configuration items and the task memory model corresponding to the task, determining the memory to be verified of at least part of the memory area in the task memory model; verifying the memory to be verified and generating a verification log, and managing the task based on the verification log; wherein, when the memory to be verified of all the obtained memory areas has passed the verification, the task can be submitted for execution.

[0005] To solve the above technical problems, the second aspect of the present application provides an electronic device, which includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method described in the first aspect.

[0006] In order to solve the above technical problem, the third aspect of the present application provides a computer-readable storage medium on which program data is stored. When the program data is executed by a processor, the method described in the first aspect is implemented.

[0007] The above scheme obtains the task that has been configured and determines the configuration information configured for the task, wherein the configuration information includes at least configuration items related to memory. Based on the model architecture of the task memory model corresponding to the configuration items and the tasks, the memory of at least part of the memory area in the task memory model is detected, and the memory to be verified of the corresponding memory area is determined, so that the memory of the memory area is detected before the task is submitted for operation, so that the memory detection is pre-placed to improve the efficiency of memory verification feedback. The memory to be verified is verified and a corresponding verification log is generated, and the task is managed based on the verification log. Among them, when all the memory areas that have been obtained have passed the verification, the configured tasks are submitted for operation, so that the tasks can be submitted for operation after the memory verification is completed and the verification log with feedback of the verification results is given, thereby reducing the probability of frequent errors caused by unreasonable memory configuration after the task is submitted for operation, and improving the stability of task operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0009] Figure 1 This is a flowchart of an implementation method of the Flink task management method of this application;

[0010] Figure 2 It is a flowchart of another implementation method of the Flink task management method of the present application;

[0011] Figure 3 It is a structural diagram of an implementation method of a task manager memory model of the present application;

[0012] Figure 4 It is a structural diagram of an implementation method of a job manager memory model of the present application;

[0013] Figure 5 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0014] Figure 6 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different implementation methods can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0017] The Flink task management method provided in this application is used to verify and manage configured tasks, and its corresponding execution subject is a processing unit that can call tasks and perform data processing.

[0018] See also Figure 1 , Figure 1 : is a flowchart of an implementation method of the Flink task management method of the present application, the method comprising:

[0019] S101: Acquire a configured task and determine configuration information of the task; wherein the configuration information at least includes configuration items related to memory.

[0020] Specifically, a task that has been configured is acquired, and configuration information configured for the task is determined, wherein the configuration information at least includes configuration items related to memory.

[0021] In some implementation scenarios, a configured task is obtained, the task type is determined, configuration information matching the task type is extracted, and configuration items related to memory in the configuration information are determined.

[0022] In some implementation scenarios, the configured tasks and the configuration information carried by the tasks are obtained, all the available configuration information is saved, and the configuration items related to the memory in the configuration information are determined.

[0023] Optionally, when the configured task is obtained, the process of obtaining the configuration information is automatically triggered, and the configuration information can be obtained by the configuration receiving module, which is used to receive the configuration information of the Flink task. For the Flink JAR (Java Archive) task, the configuration information includes the dependent Jar package path, Jar package parameters, configuration items, and save point path. For the Flink SQL (Structured Query Language) task, it includes the Jar package path, Jar package parameters, configuration items, save point path, checkpoint configuration, and SQL text.

[0024] S102: Determine, based on the configuration item and the task memory model corresponding to the task, a memory to be verified in at least part of the memory area in the task memory model.

[0025] Specifically, based on the model architecture of the task memory model corresponding to the configuration items and tasks, the memory of at least part of the memory area in the task memory model is detected, and the memory to be checked in the corresponding memory area is determined, so that the memory of the memory area is detected before the task is submitted for execution, so that the memory detection is put in advance to improve the efficiency of memory verification feedback.

[0026] In some implementation scenarios, all memory areas of the task memory model correspond to mandatory configuration items. The configured memory corresponding to some memory areas is obtained by using the parameters corresponding to the configuration items. Based on the configured memory and the model architecture of the task memory model, the memory of other memory areas is inferred to obtain the estimated memory. Among them, both the configured memory and the estimated memory are used as the memory to be verified.

[0027] In some implementation scenarios, the total memory allocated for the task is obtained based on the configuration items, and based on the total memory, the model architecture of the task memory model and the configuration items, the memory areas at different levels in the task memory model are obtained step by step to determine the memory to be checked in each memory area.

[0028] It can be understood that the task memory model of a Flink task generally includes a task manager memory model and a job manager memory model, and the memory to be verified corresponding to the memory area of ​​the memory model is obtained for each of the two memory models.

[0029] S103: Verify the memory to be verified and generate a verification log, and manage the task based on the verification log; wherein, when the memory to be verified of all the acquired memory areas passes the verification, the task can be submitted for execution.

[0030] Specifically, the memory to be checked is checked and a corresponding check log is generated, and the task is managed based on the check log. When all the memory areas that have been obtained have passed the check, the configured task is submitted for operation, so that the task can be submitted for operation after the memory check is completed and the check log with feedback of the check result is given, which reduces the probability of frequent errors caused by unreasonable memory configuration after the task is submitted for operation, and improves the stability of task operation.

[0031] In some implementation scenarios, a verification method matching the memory area is obtained, and the memory to be verified in the memory area is verified using the corresponding verification method to generate a verification log. The verification log includes the memory to be verified in the memory area, the verification method, and the verification result. If the verification of all the memory to be verified in the obtained memory area passes, the task can be submitted for execution.

[0032] In some implementation scenarios, a reference memory interval corresponding to a memory area is obtained, and the memory to be verified in the memory area is compared with the reference memory interval to obtain the verification result of the memory to be verified and generate a verification log. The verification log includes the memory to be verified and the verification result of the memory area. If the verification of all the obtained memory areas to be verified passes, the task can be submitted for execution.

[0033] It is understandable that when the verification result is failed, the verification process is terminated, and corresponding prompt information can be generated after the verification process is terminated.

[0034] The above scheme obtains the task that has been configured and determines the configuration information configured for the task, wherein the configuration information includes at least configuration items related to memory. Based on the model architecture of the task memory model corresponding to the configuration items and the tasks, the memory of at least part of the memory area in the task memory model is detected, and the memory to be verified of the corresponding memory area is determined, so that the memory of the memory area is detected before the task is submitted for operation, so that the memory detection is pre-placed to improve the efficiency of memory verification feedback. The memory to be verified is verified and a corresponding verification log is generated, and the task is managed based on the verification log. Among them, when all the memory areas that have been obtained have passed the verification, the configured tasks are submitted for operation, so that the tasks can be submitted for operation after the memory verification is completed and the verification log with feedback of the verification results is given, thereby reducing the probability of frequent errors caused by unreasonable memory configuration after the task is submitted for operation, and improving the stability of task operation.

[0035] See also Figure 2 , Figure 2 : is a flowchart of another implementation of the Flink task management method of the present application, the method comprising:

[0036] S201: Acquire a configured task and determine configuration information of the task; wherein the configuration information at least includes configuration items related to memory.

[0037] Specifically, a task that has been configured is acquired, and configuration information configured for the task is determined, wherein the configuration information at least includes configuration items related to memory.

[0038] S202: Determine, based on the configuration item and the task memory model corresponding to the task, a memory to be verified in at least part of the memory area in the task memory model.

[0039] Specifically, the task memory model includes a task manager memory model and a job manager memory model.

[0040] It can be understood that determining the memory to be checked of at least part of the memory area in the task memory model based on the task memory model corresponding to the configuration item and the task includes: obtaining the total task memory of the task manager memory model in the configuration item, and determining the memory to be checked of at least part of the memory area in the task manager memory model based on the total task memory and the model architecture corresponding to the task manager memory model; and obtaining the total job memory of the job manager memory model in the configuration item, and determining the memory to be checked of at least part of the memory area in the job manager memory model based on the total job memory and the model architecture corresponding to the job manager memory model.

[0041] Specifically, see Figure 3 and Figure 4 , Figure 3 It is a structural diagram of an implementation method of the task manager memory model of the present application. Figure 4 It is a structural diagram of an implementation method of the job manager memory model of the present application.

[0042] It can be understood that the total task memory configured for the task manager memory model is obtained from the configuration item, and based on the model architecture corresponding to the total task memory and the task manager memory model, the memory of at least part of the memory area in the task manager memory model is detected to obtain the memory to be checked in the corresponding memory area.

[0043] Similarly, the total job memory configured for the job manager memory model is obtained from the configuration item, and based on the total job memory and the model architecture corresponding to the job manager memory model, the memory of at least part of the memory area in the job manager memory model is detected to obtain the memory to be verified of the corresponding memory area. Therefore, by using the total memory and model architecture indicated by the configuration item, the memory area in the task manager memory model and the memory area in the job manager memory model are detected respectively, and the memory can be verified before the task is submitted for execution.

[0044] It should be noted that a check is performed each time the memory to be checked in any memory area is obtained, and various check methods are matched with the memory to be checked in each memory area. When the memory to be checked in the memory area passes the check, the check log records the memory to be checked in the memory area. When the memory to be checked in the memory area fails the check, the check process is terminated and the check log records the matching check method and check process.

[0045] Specifically, each memory area's memory to be checked is matched with various verification methods. When the memory to be checked in the memory area is obtained, the corresponding verification method can be used for verification, so as to obtain the verification result of the memory to be checked as soon as possible. When the verification passes, the value of the memory to be checked in the memory area is recorded in the verification log, and the verification process can be terminated in time when the verification fails. The verification method and verification process are recorded in the verification log to show the reason why the verification failed.

[0046] It is understandable that the verification methods matching different types of memory areas include but are not limited to verifying whether they are configured, verifying whether they are between the minimum and maximum values, and verifying the memory configuration ratio, so as to adapt to different types of memory areas.

[0047] In some implementation scenarios, the total task memory of the task manager memory model in the configuration item is obtained, and based on the total task memory and the model architecture corresponding to the task manager memory model, the memory to be checked of at least part of the memory area in the task manager memory model is determined, including: obtaining a configuration value in the configuration item that matches the total task memory as the total task memory; determining the metaspace memory and running overhead memory of the task manager memory model in the configuration item based on the model architecture corresponding to the task manager memory model, and determining the task-associated memory of the task manager memory model based on the total task memory, the metaspace memory and the running overhead memory of the task manager memory model; obtaining the configuration parameters associated with the task-associated memory in the configuration item, and determining the task heap memory, managed memory and network memory included in the task-associated memory based on the model architecture and configuration parameters corresponding to the task manager memory model.

[0048] Specifically, obtain the configuration value that matches the total memory of the task in the configuration item, that is, the total process memory taskmanager.memory.process.size, as the total memory of the task, where the total memory of the task is recorded as totalProcessMemorySize. Based on the model architecture corresponding to the task manager memory model, obtain the metaspace memory jvmMetaspaceSize of the task manager memory model from the configuration item, and obtain the maximum, minimum and configuration ratio of the running overhead memory JvmOverhead of the task manager memory model. The minimum value of the verification must be less than the maximum value, and the configuration ratio size must be between (0, 1). If not satisfied, the verification fails and the verification log is recorded. If satisfied, continue to calculate jvmOverheadSize, and the jvmOverheadSize size must be less than totalProcessMemorySize.

[0049] Furthermore, if the metaspace memory and the runtime overhead memory check pass, the task associated memory of the task manager memory model is determined based on the total task memory, the metaspace memory of the task manager memory model, and the runtime overhead memory. Here, the task associated memory totalFlinkMemory=totalProcessMemorySize-totalJvmMetaspaceAndOverheadSize.

[0050] It can be understood that based on the model architecture of the task manager memory model, the task-associated memory is composed of multiple memory areas. The configuration parameters associated with the task-associated memory in the configuration items are obtained. Based on the model architecture and configuration parameters corresponding to the task manager memory model, the task heap memory taskHeapMemorySize, managed memory managedMemorySize and network memory networkMemorySize included in the task-associated memory can be detected to obtain the memory to be checked in multiple memory areas in the task manager memory model, so that the memory verification has higher accuracy.

[0051] In a specific implementation scenario, configuration parameters associated with task-associated memory in a configuration item are obtained, and based on the model architecture and configuration parameters corresponding to the task manager memory model, the task heap memory, managed memory, and network memory included in the task-associated memory are determined, including: in response to the configuration parameter including a configuration value of the task heap memory, the task heap memory is obtained, and the managed memory and network memory are determined based on the model architecture and the task heap memory corresponding to the task manager memory model; in response to the configuration parameter including a configuration ratio of managed memory, the managed memory is determined based on the task-associated memory and the configuration ratio, and the network memory and task heap memory are determined based on the model architecture and the managed memory corresponding to the task manager memory model.

[0052] Specifically, determine whether the configuration item configures the configuration value of the task heap memory, that is, determine whether it includes the configuration item taskmanager.memory.task.heap.size. When the configuration parameter includes the configuration value of the task heap memory, obtain the task heap memory taskHeapMemorySize, and calculate the managed memory managedMemorySize and network memory networkMemorySize based on the model architecture and task heap memory corresponding to the task manager memory model. Verify networkMemorySize and verify that the total memory of each memory area is less than the configured total memory value. If not, the verification fails and the verification log is recorded.

[0053] Furthermore, when the configuration parameters do not include the configuration value of the task heap memory, obtain the configuration ratio of the managed memory included in the configuration parameters, calculate the managed memory managedMemorySize based on the task-associated memory and the configuration ratio, and calculate the network memory networkMemorySize and the task heap memory taskHeapMemorySize based on the model architecture and managed memory corresponding to the task manager memory model. Verify that the total memory of each memory area is less than the configured total memory value. If not satisfied, the verification fails and a verification log is recorded. Therefore, based on the different parameters included in the configuration items, the various types of memory included in the task-associated memory can be refined to make the memory verification more refined.

[0054] It should be noted that when calculating the network memory networkMemorySize, it is necessary to obtain the maximum and minimum values ​​and the ratio of networkMemorySize. The minimum value of the verification must be smaller than the maximum value, and the ratio must be between (0, 1). If not satisfied, the verification fails and the verification log is recorded. Calculate networkMemorySize (totalFlinkMemorySize × fraction) by configuring the ratio.

[0055] Furthermore, when verifying the network memory networkMemorySize, obtain the maximum, minimum, and ratio of the network memory (taskmanager.memory.network.min, taskmanager.memory.network.max, taskmanager.memory.network.fraction) to determine whether the inferred network memory is between the maximum and minimum values ​​of networkMemorySize. If not, the verification fails and a log is recorded. If the network memory ratio taskmanager.memory.network.fraction is configured, verify whether derivedNetworkMemory is equal to totalFlinkMemorySize × fraction (ratio). If not, the verification fails and a log is recorded.

[0056] In some implementation scenarios, the total job memory of the job manager memory model in the configuration item is obtained, and based on the total job memory and the model architecture corresponding to the job manager memory model, the memory to be verified of at least part of the memory area in the job manager memory model is determined, including: obtaining a configuration value in the configuration item that matches the total job memory as the total job memory; determining the metaspace memory and running overhead memory of the job manager memory model in the configuration item based on the model architecture corresponding to the job manager memory model, and determining the job-associated memory of the job manager memory model based on the total job memory, the metaspace memory and the running overhead memory of the job manager memory model.

[0057] Specifically, obtain the configuration value that matches the total memory of the job in the configuration item, that is, the total process memory jobmanager.memory.process.size, as the total memory of the job, where the total memory of the job is recorded as totalProcessMemorySize. Based on the model architecture corresponding to the job manager memory model, obtain the metaspace memory jvmMetaspaceSize of the job manager memory model from the configuration item, and obtain the maximum value, minimum value and configuration ratio of the running overhead memory JvmOverhead of the job manager memory model. The minimum value of the verification must be less than the maximum value, and the configuration ratio size must be between (0, 1). If not satisfied, the verification fails and the verification log is recorded.

[0058] It can be understood that the operating overhead memory JvmOverheadSize (totalProcessMemorySize × fraction) is calculated by configuring the ratio. Check whether JvmOverheadSize is less than totalProcessMemorySize. If not, the check fails and the check log is recorded.

[0059] Furthermore, if the metaspace memory and the runtime overhead memory check pass, the job-associated memory of the job manager memory model is determined based on the total job memory, the metaspace memory of the job manager memory model, and the runtime overhead memory. In which, the job-associated memory totalFlinkMemorySize=totalProcessMemory-totalJvmMetaspaceAndOverheadSize.

[0060] Optionally, obtain the off-heap memory offHeapMemorySize based on the configuration item. Verify whether totalFlinkMemorySize is greater than offHeapMemory. If not, the verification fails and the verification log is recorded. If satisfied, infer the on-heap memory JvmHeapMemorySize (totalFlinkMemorySize-offHeapMemory). Verify whether JvmHeapMemorySize is greater than the required minimum value. If not, the verification fails and the log is printed.

[0061] It can be understood that based on the model architecture and configuration parameters corresponding to the job manager memory model, the memory to be verified of multiple memory areas in the job manager memory model can be obtained, so that the memory verification has higher accuracy.

[0062] S203: Verify the memory to be verified and generate a verification log, and manage the task based on the verification log; wherein, when the memory to be verified of all the acquired memory areas passes the verification, the task can be submitted for execution.

[0063] Specifically, the memory to be verified is verified and a corresponding verification log is generated, and the task is managed based on the verification log. When all the obtained memory areas have passed the verification, the configured task is submitted for execution.

[0064] S204: In response to obtaining the suggested generation requirement, a task topology matching the task information is generated, and all task vertices in the task topology and configuration resources matching each task vertex are acquired.

[0065] Specifically, when a suggestion generation requirement is obtained, a task topology JobGraph matching the task information is generated, and the number of task vertices JobVertex of the task, the number of tasks in each JobVertex, the number of task manager slots TaskManager slots, the resource configuration of TaskManager / JobManager, and the state backend StateBackend configuration are obtained through JobGraph.

[0066] Optionally, the suggestion generation requirement may be triggered after obtaining the configuration information of the task, and the triggering method may be actively triggered by waiting for the use object, or passively triggered after completing the memory check.

[0067] In some implementation scenarios, the entire method flow is implemented based on a task management system, which includes a configuration receiving module, a configuration optimization module, a configuration verification module, and a configuration sending module. When an object is used to trigger a memory optimization operation, the configuration receiving module receives the configuration information of the task, and after analysis by the configuration optimization module, outputs configuration optimization suggestions, which are pushed to the terminal for display by the configuration sending module. When an object is used to trigger a memory verification operation, the configuration receiving module receives the configuration information of the task, and after verification by the configuration verification module, outputs the verification log and verification results, which are pushed to the terminal for display by the configuration sending module.

[0068] S205: Generate resource configuration suggestions based on the configuration resources matched by the task vertices.

[0069] Specifically, based on the configuration resources matched by the task vertex, a configuration suggestion matching the current configuration of the task is given, wherein the configuration suggestion corresponds to a matching configuration strategy.

[0070] In some implementation scenarios, the specific configuration strategy is: If the resource configuration of TaskManager is less than 1C 4G (1 4G CPU), it is recommended to adjust the resources to 1C 4G. If the resource configuration of jobManager is less than 1C 4G, it is recommended to adjust the resources to 1C 4G. If the number of slots of a single TaskManager is greater than 1, it is recommended to adjust the resources to (1C4G) × the number of slots of a single TaskManager. If the State Backend is not the RocksDB State Backend, it is recommended to set the managed memory to 0, that is, taskmanager.memory.managed.size=0. If there are more than 20 tasks under a slotSharingGroup, it is prompted that more than 20 tasks will be run under a slot slot. It is recommended to use the object to increase the resource configuration of TaskManager during configuration. Among them, in different implementation scenarios, different configuration strategies can customize other specific configuration values, and this application does not make specific restrictions on this.

[0071] In a specific implementation scenario, it is recommended that generation requirements be triggered after all acquired memory areas to be verified have passed verification; based on the configuration resources matched by the task vertices, resource configuration suggestions are generated, including: based on the configuration resources matched by the task vertices and all acquired memory areas to be verified, resource configuration suggestions are generated.

[0072] Specifically, when the requirement for suggestion generation is triggered after the memory to be verified of all acquired memory areas has passed, then when giving resource configuration suggestions, it is also possible to generate resource configuration suggestions based on the configuration resources matched by the task vertices and the memory to be verified of the acquired memory areas, so that the resource configuration suggestions have higher accuracy.

[0073] In this embodiment, the task memory model includes the task manager memory model and the job manager memory model. The total memory and model architecture indicated by the configuration item are used to detect the memory area in the task manager memory model and the memory area in the job manager memory model respectively, so that the memory can be verified before the task is submitted for execution. In addition to verifying the memory, a configuration suggestion matching the current configuration of the task can also be given based on the configuration resources matched by the task vertex.

[0074] See also Figure 5 , Figure 5It is a structural diagram of an embodiment of an electronic device of the present application, wherein the electronic device 50 includes a memory 501 and a processor 502 coupled to each other, wherein the memory 501 stores program data (not shown), and the processor 502 calls the program data to implement the method in any of the above embodiments. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0075] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 60 stores program data 600. When the program data 600 is executed by a processor, the method in any of the above embodiments is implemented. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0076] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0077] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0079] The above description is only an implementation method of the present application, and does not limit the protection scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the protection scope of the present application.

Claims

1. A Flink task management method, characterized in that: The method comprises: Acquire a configured task and determine configuration information of the task; wherein the configuration information at least includes configuration items related to memory; Based on the configuration item and the task memory model corresponding to the task, determine the memory to be checked of at least part of the memory area in the task memory model; wherein the configuration item indicates the total memory and model architecture of the task memory model, and the model architecture corresponds to multiple memory areas, and the configuration item includes configuration parameters of some memory areas; verify the memory to be checked and generate a verification log, and manage the task based on the verification log; wherein, when the memory to be checked of all the obtained memory areas has passed the verification, the task can be submitted for execution.

2. The Flink task management method according to claim 1, characterized in that: The task memory model includes a task manager memory model and a job manager memory model, and determining the memory to be verified of at least part of the memory area in the task memory model based on the configuration item and the task memory model corresponding to the task includes: Obtaining the total task memory of the task manager memory model in the configuration item, and determining the memory to be checked of at least part of the memory area in the task manager memory model based on the total task memory and the model architecture corresponding to the task manager memory model; and The total job memory of the job manager memory model in the configuration item is obtained, and based on the total job memory and a model architecture corresponding to the job manager memory model, a memory to be verified of at least part of the memory area in the job manager memory model is determined.

3. The Flink task management method according to claim 2, characterized in that: The obtaining the total task memory of the task manager memory model in the configuration item, and determining the memory to be checked of at least part of the memory area in the task manager memory model based on the total task memory and the model architecture corresponding to the task manager memory model, comprises: Obtaining a configuration value in the configuration item that matches the total memory of the task as the total memory of the task; Determine the metaspace memory and the running overhead memory of the task manager memory model in the configuration item based on the model architecture corresponding to the task manager memory model, and determine the task associated memory of the task manager memory model based on the total task memory, the metaspace memory and the running overhead memory of the task manager memory model; The configuration parameters associated with the task-associated memory in the configuration item are obtained, and based on the model architecture corresponding to the task manager memory model and the configuration parameters, the task heap memory, the managed memory, and the network memory included in the task-associated memory are determined.

4. The Flink task management method according to claim 3, characterized in that: The obtaining of the configuration parameters associated with the task-associated memory in the configuration item, and determining the task heap memory, managed memory, and network memory included in the task-associated memory based on the model architecture corresponding to the task manager memory model and the configuration parameters, comprises: In response to the configuration parameter including a configuration value of the task heap memory, acquiring the task heap memory, and determining the managed memory and the network memory based on a model architecture corresponding to the task manager memory model and the task heap memory; In response to the configuration parameters including the configuration ratio of the managed memory, the managed memory is determined based on the task-associated memory and the configuration ratio, and the network memory and the task heap memory are determined based on the model architecture corresponding to the task manager memory model and the managed memory.

5. The Flink task management method according to claim 2, characterized in that: The obtaining of the total job memory of the job manager memory model in the configuration item, and determining the memory to be verified of at least part of the memory area in the job manager memory model based on the total job memory and the model architecture corresponding to the job manager memory model, comprises: Obtaining a configuration value in the configuration item that matches the total memory of the job as the total memory of the job; Based on the model architecture corresponding to the job manager memory model, the metaspace memory and running overhead memory of the job manager memory model in the configuration item are determined; based on the total job memory, the metaspace memory and running overhead memory of the job manager memory model, the job associated memory of the job manager memory model is determined.

6. The Flink task management method according to any one of claims 1 to 5, characterized in that: A check is performed each time the memory to be checked of any of the memory areas is obtained, and each memory to be checked of the memory area is matched with various check methods. When the memory to be checked of the memory area passes the check, the check log records the memory to be checked of the memory area; when the memory to be checked of the memory area fails the check, the check process is terminated and the check log records the matching check method and check process.

7. The Flink task management method according to claim 1, characterized in that: The configuration information also includes task information related to the task, and the method further includes: In response to obtaining the suggested generation requirement, generating a task topology matching the task information, and acquiring all task vertices in the task topology and configuration resources matching each of the task vertices; Generate resource configuration suggestions based on the configuration resources matched by the task vertices.

8. The Flink task management method according to claim 7, characterized in that: The suggestion generation requirement is triggered after all the obtained memory areas to be verified have passed the verification; The configuring resources based on the task vertex matching, generating resource configuration suggestions, includes: Generate resource configuration suggestions based on the configuration resources matched by the task vertex and all the obtained memory to be verified in the memory area.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images