Debugging processing method and device for real-time operation through computing power of intelligent computing center

Abstract

The embodiment of the invention provides a debugging processing method and device for real-time operation through computing power of an intelligent computing center. The method comprises the following steps: S1, acquiring a target operator selected by a user on a front-end editing interface; s2, performing job debugging operation based on the target operator; and S3, displaying an output result of the job debugging operation on the front-end editing interface in real time. According to the embodiment of the invention, the user can be allowed to flexibly select the required operator as the debugging operator on the interface, and the resource occupation of the debugging operation can be reduced when the computing engine resources are limited.

Description

Technical Field

[0001] The present invention relates to the field of computing power infrastructure, and in particular to a debugging processing method and device for real-time computing operations through intelligent computing center computing power. Background Art

[0002] In recent years, with the development of artificial intelligence technology and computing power technology, the concept of intelligent computing center has emerged. "Intelligent computing center" refers to the use of large-scale heterogeneous computing power resources, including general computing power and intelligent computing power, mainly for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning and other scenarios) to provide the required computing power, data and algorithms. Intelligent computing center covers facilities, hardware, software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement. On the other hand, with the rapid development of big data, all walks of life have begun to apply big data processing and analysis. Big data processing and analysis can use the intelligent computing center to create jobs to achieve data processing and analysis. Flink is a data processing framework that can perform stream processing and batch processing. In Flink, a job can be a data processing flow composed of one or more operators. Each operator is a data processing unit that is responsible for receiving input data streams, performing data transformation and calculations, and then passing the output data stream to the next operator. Operators are divided into three types: source operators, transform operators, and sink operators. The source operator is responsible for reading data from different types of external storage systems, the conversion operator can perform various conversions on the data, and the sink operator is responsible for outputting the data to different types of external storage systems. Flink provides a Web UI that can monitor the status, logs, data flow processing speed and other information of the running job, but it cannot view the processing results of the actual data in the job. In the prior art, the important intermediate operators in the tested job or other operators except the last sink operator are spliced ​​with the dump operator, and then the test data is input into the external storage system connected to the source operator, and the job is submitted to Flink for operation. The dump operator will store the output results of the connected operators in the database, and finally the data display platform will query the database for the intermediate test results stored by the dump operator, and query the final result of the entire job in the external storage system connected to the sink operator. In the prior art, by adding a dump operator to view the processing results of the actual data in the job, it is impossible to flexibly select the debug object operator for intermediate result output, and the user cannot flexibly adjust the input test data and compare the test results at the same time during the debugging process, and may cause the resources used by the job to exceed the limit; additional storage requirements are added. Summary of the invention

[0003] The embodiments of the present invention provide a method and device for debugging and processing real-time jobs through the computing power of an intelligent computing center, which solves the problem that in the prior art, a dump operator is added to view the processing results of actual data in the job, but the method cannot flexibly select the debugging object operator to output intermediate results. The user cannot flexibly adjust the input test data and compare the test results at the same time during the debugging process, and may cause the resources used by the job to exceed the limit, adding additional storage requirements.

[0004] In order to solve the above technical problems, the present invention provides a debugging processing method for real-time operation through intelligent computing center computing power, the method comprising:

[0005] Step S1: Obtain the target operator selected by the user in the front-end editing interface;

[0006] Step S2: performing job debugging and operation based on the target operator;

[0007] Step S3: Displaying the output result of the job debugging operation in real time on the front-end editing interface.

[0008] Optionally, in the above method, obtaining the target operator selected by the user in the front-end editing interface includes:

[0009] Step S11: displaying the initial job directed acyclic graph on the front-end editing interface;

[0010] Step S12: Obtain the target operator to be debugged and run selected by the user from the initial job directed acyclic graph.

[0011] Optionally, in the above method, the performing job debugging and running based on the target operator includes:

[0012] Step S21: calling the debugging mode to start the job based on the user's target operation;

[0013] Step S22: preprocessing the initial job directed acyclic graph based on the target operator to obtain a target job directed acyclic graph;

[0014] Step S23: Debugging and running the target job directed acyclic graph.

[0015] Optionally, in the above method, preprocessing the initial job directed acyclic graph based on the target operator to obtain the target job directed acyclic graph includes:

[0016] Step S221: modifying the initial job directed acyclic graph based on the target operator to obtain the target job directed acyclic graph.

[0017] Optionally, in the above method, the target job directed acyclic graph is obtained by modifying the initial job directed acyclic graph in any of the following ways:

[0018] Step S2211: replacing the source operator of the initial job directed acyclic graph with the Websocket source operator; adding a new Websocket sink operator after the target operator of the initial job directed acyclic graph; replacing the sink operator of the initial job directed acyclic graph with the Websocket sink operator;

[0019] Step S2212: Add a Websocket sink operator after the source operator of the initial job directed acyclic graph; add a Websocket source operator before the successor operator of the source operator of the initial job directed acyclic graph; add a new Websocket sink operator after the target operator of the initial job directed acyclic graph; replace the sink operator of the initial job directed acyclic graph with the Websocket sink operator.

[0020] Optionally, before debugging and running the target job directed acyclic graph, the method further includes:

[0021] Step S24: re-estimating the resources occupied by the target job according to the number of the Websocket source operators in the directed acyclic graph of the target job and the number of the Websocket sink operators included in the directed acyclic graph of the target job;

[0022] Step S23: taking the target job occupied resources as target parameters for submitting the computing engine job.

[0023] Optionally, in the above method, the debugging and running of the target job directed acyclic graph includes:

[0024] Step S231: submitting the target job directed acyclic graph to a computing engine according to the target parameters;

[0025] Step S232: Run the target job directed acyclic graph through the computing engine to obtain an output result of the job debugging operation.

[0026] The embodiment of the present invention further provides a debugging and processing device for real-time operation of computing power in an intelligent computing center, the device comprising:

[0027] The acquisition module is used to obtain the target operator selected by the user in the front-end editing interface;

[0028] A processing module, used for debugging and running a job based on the target operator;

[0029] A display module is used to display the output result of the job debugging operation in real time on the front-end editing interface.

[0030] Optionally, in the above device, the acquisition module includes:

[0031] A display submodule, used for displaying the initial job directed acyclic graph on the front-end editing interface;

[0032] The first acquisition submodule is used to acquire the target operator to be debugged and run selected by the user from the initial job directed acyclic graph.

[0033] Optionally, in the above device, the processing module includes:

[0034] A startup submodule, used to call a debug mode startup job based on a user's target operation;

[0035] A preprocessing submodule, used for preprocessing the initial job directed acyclic graph based on the target operator to obtain a target job directed acyclic graph;

[0036] The processing submodule is used to debug and run the target job directed acyclic graph.

[0037] Optionally, in the above device, the preprocessing submodule includes:

[0038] The modification unit is used to modify the initial job directed acyclic graph based on the target operator to obtain the target job directed acyclic graph.

[0039] Optionally, in the above device, the target job directed acyclic graph is obtained by modifying the initial job directed acyclic graph in any of the following ways:

[0040] Replacing the source operator of the initial job directed acyclic graph with the Websocket source operator; adding a new Websocket sink operator after the target operator of the initial job directed acyclic graph; replacing the sink operator of the initial job directed acyclic graph with the Websocket sink operator;

[0041] Add a Websocket sink operator after the source operator of the initial job directed acyclic graph; add a Websocket source operator before the successor operator of the source operator of the initial job directed acyclic graph; add a new Websocket sink operator after the target operator of the initial job directed acyclic graph; replace the sink operator of the initial job directed acyclic graph with the Websocket sink operator.

[0042] Optionally, the above device further comprises:

[0043] An estimation submodule, used for re-estimating the resources occupied by the target job according to the number of the Websocket source operators in the directed acyclic graph of the target job and the number of the Websocket sink operators included in the directed acyclic graph of the target job;

[0044] The submitting submodule is used to use the resources occupied by the target job as target parameters for submitting the computing engine job.

[0045] Optionally, in the above device, the processing submodule includes:

[0046] A submitting unit, configured to submit the target job directed acyclic graph to a computing engine according to the target parameters;

[0047] The processing unit is used to run the directed acyclic graph of the target job through the computing engine to obtain the output result of the job debugging operation.

[0048] An embodiment of the present invention also provides a debugging and processing system for real-time operations of computing power in an intelligent computing center, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the real-time operation processing method as described above when executed by the processor.

[0049] An embodiment of the present invention further provides a readable storage medium, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned real-time job processing method when executed by the processor.

[0050] The embodiment of the present invention can allow the user to flexibly select a required operator as a debugging operator on the interface, and can also reduce the occupation of resources by debugging operations when computing engine resources are limited.

[0051] Furthermore, the embodiments of the present invention provide users with a convenient, easy-to-use and generally applicable Flink operator job full-process monitoring and debugging method for various operators. Users can debug Flink operator jobs in the same interface without modifying the original jobs and operators, and quickly locate problematic operators through real-time output data of key operators, and debug again after rapid modification. After successful debugging, the job can be officially enabled without any changes, thereby improving job development efficiency and reducing storage requirements.

[0052] Furthermore, the embodiment of the present invention avoids complex background implementation by replacing or adding a Websocket operator to send and receive real-time data, and the debugging interface is simple and easy to understand; during the debugging process, the user can choose to use the external storage system data of the original source operator of the job as the input test data to verify the rationality of the original data, or manually input any data as the input test data to flexibly cover various situations; the user can select any operator (including various types of source operators) except the host operator to view its output results, that is, it can reduce the interference of irrelevant data when observing the test results, and can also reduce the resources occupied by the debugging job when Flink resources are limited; and the implementation of the present invention does not actually connect the results to the host operator during debugging, thereby avoiding data pollution to the external storage system connected to the host operator, and further facilitating users to debug and quickly locate problems in the production environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0054] Figure 1 It is a flowchart of a debugging processing method for real-time operation through intelligent computing center computing power provided by an embodiment of the present invention;

[0055] Figure 2 It is a schematic diagram of an intelligent computing center front-end editing interface provided by an embodiment of the present invention;

[0056] Figure 3 It is a schematic diagram of another intelligent computing center front-end editing interface provided by an embodiment of the present invention;

[0057] Figure 4 It is a timing diagram of a debugging job initiated by a user of an intelligent computing center provided by an embodiment of the present invention;

[0058] Figure 5 It is a timing diagram of debugging operation period through intelligent computing center computing power operation provided by an embodiment of the present invention;

[0059] Figure 6 It is a schematic diagram of another intelligent computing center front-end editing interface provided by an embodiment of the present invention;

[0060] Figure 7 It is a schematic diagram of another intelligent computing center front-end editing interface provided by an embodiment of the present invention;

[0061] Figure 8It is a directed acyclic DAG graph provided by an embodiment of the present invention;

[0062] Fig. 9 It is a schematic diagram of another intelligent computing center front-end editing interface provided by an embodiment of the present invention;

[0063] Fig.10 It is a schematic diagram of another intelligent computing center front-end editing interface provided by an embodiment of the present invention;

[0064] Fig.11 It is a structural block diagram of a debugging and processing device for real-time operation of computing power in an intelligent computing center provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The "computing power" mentioned in the present invention is the ability of computer equipment or computing / data center to process information. It is the ability of computer hardware and software to work together to execute certain computing requirements. It is the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to the society through computing power infrastructure.

[0067] The "computing power (CP)" mentioned in the present invention is the ability of a data center server to process data and output results. It is a comprehensive indicator to measure the computing power of a data center, including general computing power, super computing power and intelligent computing power. The commonly used unit of measurement is the number of floating point operations per second (Floating Point Operations Per Second, FLOPS, where 1EFLOPS=10^18FLOPS). The larger the value, the stronger the comprehensive computing power. According to calculations, 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs (Central Processing Unit) or 2 million mainstream notebooks. The calculation formula is: CP=CP 通用 +CP 智能 +CP 超级 In the embodiment of the present application, the computing power adopts the half-precision floating point computing power (FP16) of the graphics card.

[0068] The "carrying capacity" (Network Power, NP) described in the present invention is a manifestation of the data transmission capability of computing facilities, including comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, etc. Carrying capacity involves network transmission within and between data centers, and is a comprehensive indicator for measuring network transmission scheduling capabilities. In the embodiment of the present application, carrying capacity uses video memory bandwidth. In the embodiment of the present application, storage capacity uses video memory bandwidth.

[0069] The "storage power" (SP) described in the present invention refers to the comprehensive capabilities of a data center in terms of data storage capacity, performance, safety and reliability, and green and low-carbon. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and server built-in storage devices. The commonly used unit of measurement for storage capacity is exabyte (EB, 1EB = 2^60bytes), and the commonly used unit of measurement for performance is the number of reads and writes per second per unit capacity (IOPS / TB, Input / Output Operations Per Second / TB). The disaster recovery ratio is an important manifestation of safety and reliability. In the embodiment of the present application, the storage power adopts the video memory capacity.

[0070] The "computing power infrastructure" described in the present invention is a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity. It can realize centralized computing, storage, transmission and application of information, and presents characteristics such as diversity and ubiquity, intelligence and agility, safety and reliability, green and low-carbon. It is of great significance to promote industrial transformation and upgrading, enable my country's scientific and technological innovation, meet people's better life and realize high-efficiency social governance.

[0071] The “computing power” mentioned in the present invention includes general computing power CP 通用 、Intelligent computing power CP 智能 And super computing power CP 超级 .

[0072] The "general computing power" mentioned in the present invention refers to the computing power provided by servers based on CPU chips, which is used to support basic general computing such as cloud computing and edge computing.

[0073] The "intelligent computing power" mentioned in the present invention refers to the large-scale deployment of intelligent computing centers based on special chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit) for various innovative artificial intelligence applications, such as natural language processing, machine vision, etc.

[0074] The "super computing power" mentioned in the present invention is mainly the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for calculations in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0075] The "intelligent computing center" mentioned in the present invention refers to a facility that provides the required computing power, data and algorithms for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model reasoning) by using large-scale heterogeneous computing resources, including general computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.). The intelligent computing center covers facilities, hardware, and software, and can provide full-stack capabilities from bottom-level computing power to top-level application enablement.

[0076] The "computing resources" mentioned in the present invention refer to the technologies and facilities with information calculation, transmission, storage and application capabilities required for the development of the digital society, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guarantee resources such as wind, fire, water and electricity.

[0077] See also Figure 1 , Figure 1 It is a flowchart of a debugging processing method for real-time operation through intelligent computing center computing power provided by an embodiment of the present invention. The debugging processing method for real-time operation is applied to the debugging processing system for real-time operation of intelligent computing center computing power. Figure 1 As shown, the debugging processing method for real-time operation through intelligent computing center computing power includes the following steps:

[0078] Step S1, obtaining the target operator selected by the user in the front-end editing interface.

[0079] The user can select the target operator to be debugged and run according to his / her needs in the front-end editing interface. The target operator can be one or more of the source operator, intermediate operator and sink operator.

[0080] As an optional implementation manner, obtaining the target operator selected by the user in the front-end editing interface includes:

[0081] Step S11: displaying the initial job directed acyclic graph on the front-end editing interface;

[0082] Step S12: Obtain the target operator to be debugged and run selected by the user from the initial job directed acyclic graph.

[0083] Specifically, the intelligent computing center computing power front-end editing interface is as follows Figure 2 As shown in the figure, the left side of the intelligent computing center front-end editing interface displays various available source operators, intermediate operators, and sink operators; the middle part of the intelligent computing center front-end editing interface is the initial job directed acyclic DAG graph of the current job. After the user clicks on the operator or edge, the corresponding description can be seen on the right side of the front-end editing interface and the relevant parameters can be set; if the user clicks the "Start Job" button at the top of the interface, the current job can be run normally. If the user clicks the "ladybug" style button at the top of the interface, the debugging mode is entered. The user can select the target operator to be debugged from the initial job directed acyclic DAG graph of the current job in the middle of the front-end editing interface.

[0084] The embodiment of the present invention can allow the user to flexibly select a required operator as a debugging operator on the interface, and can also reduce the occupation of resources by debugging operations when computing engine resources are limited.

[0085] Step S2: Debugging and running the job based on the target operator.

[0086] Specifically, the user can select the Run or Debug button in the front-end editing interface of the intelligent computing center to debug the job of the selected target operator. Figure 3 As shown in the figure, the user can select any source operator or intermediate operator as the debugged operator, and finally click the "Start Debugging" button to start the current job in debug mode.

[0087] As an optional implementation, the step S2 of performing job debugging and running based on the target operator includes:

[0088] Step S21: calling the debugging mode to start the job based on the user's target operation;

[0089] Step S22: preprocessing the initial job directed acyclic graph based on the target operator to obtain a target job directed acyclic graph;

[0090] Step S23: Debugging and running the target job directed acyclic graph.

[0091] Specifically, Figure 3 As shown, the user can click the "Start Debug" button or the "Start Run" button in the front-end editing interface of the intelligent computing center to start the current job. The initial job directed acyclic graph is preprocessed based on the target operator to be debugged and run selected by the user to obtain the target job directed acyclic graph. The target job directed acyclic graph is debugged and run.

[0092] As an implementable manner, preprocessing the initial job directed acyclic graph based on the target operator to obtain the target job directed acyclic graph includes:

[0093] Step S221: modifying the initial job directed acyclic graph based on the target operator to obtain the target job directed acyclic graph.

[0094] Specifically, the initial job directed acyclic graph is modified using the job management module to obtain the target job directed acyclic graph.

[0095] The target job directed acyclic graph is obtained by modifying the initial job directed acyclic graph in any of the following ways:

[0096] Step S2211: replacing the source operator of the initial job directed acyclic graph with the Websocket source operator; adding a new Websocket sink operator after the target operator of the initial job directed acyclic graph; replacing the sink operator of the initial job directed acyclic graph with the Websocket sink operator;

[0097] Step S2212: Add a Websocket sink operator after the source operator of the initial job directed acyclic graph; add a Websocket source operator before the successor operator of the source operator of the initial job directed acyclic graph; add a new Websocket sink operator after the target operator of the initial job directed acyclic graph; replace the sink operator of the initial job directed acyclic graph with the Websocket sink operator.

[0098] Specifically, in order to satisfy the requirement of sending the data input by the user of the front-end interface of the intelligent computing center to the computing engine Flink job in real time and returning the operator calculation results to the front-end interface in real time, the present invention chooses to use Websocket, a full-duplex communication protocol, to establish a real-time communication connection between the front-end display module and the job management module, and between the job management module and the running job on Flink. In order to replace any source operator to send real-time test data from the front end, and to output the results of any operator in the job to the front end in real time, the job management module of the present invention pre-processes the initial job directed acyclic graph DAG of the original job before submitting the original job to Flink for running:

[0099] 1. Replace the source operator in the original job with the new Websocket source operator (the predecessors of all subsequent operators of the source operator must be modified to this new input operator. If the source operator is also the debugged operator, the predecessor of this source operator of all its subsequent operators must be retained at the same time);

[0100] 2. Connect a new Websocket sink operator to all operators that need to be debugged;

[0101] 3. Replace the original sink operator with the new Websocket sink operator.

[0102] Among them, the Websocket sink operator receives the output data of the predecessor operator and sends it to the Websocket server of the job management module; the Websocket source operator receives the test data sent by the Websocket server of the job management module and passes it to its successor operator.

[0103] In addition, the source operator may be retained, and a Websocket sink operator may be added after the source operator, and a Websocket source operator may be added before the original successor operator of the source operator.

[0104] As an implementable manner, before debugging and running the target job directed acyclic graph, the method further includes:

[0105] Step S24: re-estimating the resources occupied by the target job according to the number of the Websocket source operators in the directed acyclic graph of the target job and the number of the Websocket sink operators included in the directed acyclic graph of the target job;

[0106] Step S25: taking the target job occupied resources as target parameters for submitting the computing engine job.

[0107] Specifically, the resources occupied by this job are re-estimated according to the number of newly added Websocket source operators and Websocket sink operators in the directed acyclic graph of the target job and used as parameters for submitting the Flink job, and the resources occupied by the target job are used as target parameters for submitting the computing engine job.

[0108] As an implementable manner, the debugging and running of the target job directed acyclic graph includes:

[0109] Step S231: submitting the target job directed acyclic graph to a computing engine according to the target parameters;

[0110] Step S232: Run the target job directed acyclic graph through the computing engine to obtain an output result of the job debugging operation.

[0111] Specifically, the job management module submits the modified target job directed acyclic graph to the computing engine Flink based on the re-estimated target job occupied resources. The computing engine Flink runs the target job directed acyclic graph to obtain the output result of the job debugging operation. Specifically, the timing of the user initiating the debugging job is as follows: Figure 4 As shown in the figure, the timing diagram of the job debugging operation period is as follows Figure 5 shown.

[0112] The embodiment of the present invention avoids complex background implementation by replacing or adding a Websocket operator to send and receive real-time data, and the debugging interface is simple and easy to understand. During the debugging process, the user can choose to use the external storage system data of the original source operator of the job as input test data to verify the rationality of the original data, or manually input any data as input test data to flexibly cover various situations. The user can select any operator (including various types of source operators) except the sink operator to view its output result, which can reduce the interference of irrelevant data when observing the test results, and can also reduce the resources occupied by the debugging job when Flink resources are limited. In addition, the implementation of the present invention does not actually connect the results to the sink operator during debugging, thereby avoiding data pollution to the external storage system connected to the sink operator, and further facilitates users to debug and quickly locate problems in the production environment.

[0113] Step S3, displaying the output result of the job debugging operation in real time on the front-end editing interface.

[0114] Specifically, Figure 6 The figure shows the front-end editing interface of the intelligent computing center computing power after the job is debugged and run. The lower part of the left column of the intelligent computing center computing power front-end editing interface is the input column for simulating the source operator to send test data, and the upper part is the debugging information output column, which can display the test data entered by the input column and the output data of all debugged operators of the current job in real time.

[0115] To achieve this process, you need to establish a Websocket server in the job management module and a Websocket client on the front-end browser and in the Websocket operator running on Flink.

[0116] The embodiment of the present invention provides users with a convenient and easy-to-use Flink operator job full-process monitoring and debugging method that is generally applicable to various operators. It allows users to debug Flink operator jobs in the same interface without modifying the original jobs and operators, and quickly locate problematic operators through real-time output data of key operators, and debug again after rapid modification. After successful debugging, the job can be officially enabled without any changes, which improves job development efficiency and reduces storage requirements. Each time a user starts a Flink operator job in debugging mode through the intelligent computing center, the present invention completes the following tasks:

[0117] 1. The front-end code establishes a Websocket client on the browser using the Websocket address of the server, and sends a message to the specified URL " / app / rcvDebugSourceMsg" of the server when the user enters debugging data. The message must contain the test data entered by the user and the operatorId that can uniquely distinguish the specified operator of the current job (this operator id can uniquely distinguish all job operators of the entire invention), and this Websocket client subscribes to the topic " / user / client" + jobId + " / queue / result" to the server, where jobId is the current job id.

[0118] 2. Each Websocket source operator running on Flink also establishes its own Websocket client with the Websocket address of the server and subscribes to the topic " / user / %operatorId% / queue / result", where operatorId is the globally unique operator ID of the source operator it replaces, to receive the front-end test data forwarded by the Websocket server;

[0119] 3. Each Websocket sink operator running on Flink also establishes a Websocket client with the Websocket address of the server, and sends a message to the specified URL " / app / rcvDebugSinkMsg" of the Websocket server. The message contains the output data of its predecessor operator and the operatorId of the predecessor operator or the operatorId of the replaced sink operator;

[0120] 4. When the Websocket server in the job management module receives a message sent to " / app / rcvDebugSourceMsg" (i.e., the test data sent from the front end), it forwards it to the " / user / %operatorId% / queue / result" topic (i.e., the Websocket source operator that replaces the source operator); when it receives a message sent to " / app / rcvDebugSinkMsg" (i.e., the output result sent from the Websocket sink operator), it forwards it to the client that subscribes to the " / user / client" + jobId + " / queue / result" topic (i.e., the front-end Websocket client of the current job).

[0121] 5. After receiving the message from the sink operator forwarded by the Websocket server, the front end distinguishes and displays which operator outputs the current test result based on the operatorId of the predecessor operator or the operatorId of the replacement sink operator in each message.

[0122] For example, Figure 7 For example, the user selects "Input format conversion 1", "JDBC source_1.13" and "Output format conversion 2" as debugging operators in the intelligent computing center and starts the job in debugging mode. The directed acyclic DAG graph after preprocessing by the job management module before submitting the job to Flink is as follows: Figure 8 As shown. Figure 7 From the data output column on the upper left column, we can see that after this job is started, since the original "JDBC source_1.13" is still connected as input, at 11:32:30 when the job is just started successfully, "JDBC source_1.13" reads two pieces of data from the database: {"name":"d","id":"00002"} and {"name":"d","id":"d"}, and forwards the result to the front end through the Websocket sink operator connected to it. Then, the Websocket sink operator connected to "Output format conversion 2" forwards the output result of "Output format conversion 2" after filtering by the edge condition to the front end. Finally, the Websocket sink operator that replaces the "Kafka sink" operator also forwards the final output result to the front end.

[0123] About 2 minutes later, at 11:34:24, the user sent a test data to the Websocket source operator that replaced "JDBC source_1.13" in the test data input bar of the interface. The Websocket sink operator connected to "Output format conversion 2" and the Websocket sink operator that replaced the "Kafka sink" operator immediately forwarded the flow results of this test data to the front end.

[0124] like Fig. 9 As shown, at 11:36:22, the user sent a piece of data that meets the filter conditions on the left to the Websocket source operator that replaces the "Kafka source" from the interface test data input bar, and forwarded it to the front end through the Websocket sink operator connected by "Input format conversion 1" and "Output format conversion 2" and the Websocket sink operator that replaces the "Kafka sink", allowing the user to see the test results of this test data after each debugging operator and the final output result.

[0125] After the above operations, users can directly view the MySql table connected to the JDBC sink and the Kafka topic connected to the Kafka sink, and cannot see any data entering, because these sink operators are not connected to the actual DAG graph.

[0126] Fig.10 This is an example of debugging an operator job that associates input data based on a time window, which fully demonstrates the advantages of the present invention in real-time debugging of time-sensitive operator jobs. In this example, the watermark time is the event processing time, and it increases with the system time. At 13:40:03, the user sent a piece of data {"type":"left","id":"1"} that meets the left filter condition to the Websocket source operator that replaces "KafkaSource" through the test data input bar on the lower left side of the interface. The Websocket sink operator behind the debugged operator "watermark 1" immediately outputs this data. Then, at 13:40:10, the user sent a piece of data {"type":"right","id":"1"} that meets the right filter condition through the same method. The Websocket sink operator behind the debugged operator "watermark 2" immediately outputs this data. Then, at 13:40:11, it can be seen that the Websocket sink operator that replaces "KafkaSink" passes these two input data through the "window "join" uses 10-second intervals as the rolling window and field id as the association field to perform full association and combination of them {"left.type":"left","right.type":"right","right.id":"1","left.id":"1"} and forwards it to the front end; in the next 10-second window ending at 13:40:21, the user only inputs the test data {"type":"left","id":"1"} once at 13:40:17, so when the window time expires, the job finally outputs a combined data {"left.type":"left","left.id":"1"} containing only the left data.

[0127] Based on the method for processing real-time operations through intelligent computing center computing power provided in the above embodiment, the embodiment of the present invention also provides a processing device for real-time operations of intelligent computing center computing power implementing the above method, please refer to Fig.11 An embodiment of the present invention provides a debugging and processing device 1100 for real-time operation of computing power in an intelligent computing center. The debugging and processing device 1100 for real-time operation of computing power in an intelligent computing center includes:

[0128] The acquisition module 1101 is used to acquire the target operator selected by the user in the front-end editing interface;

[0129] A processing module 1102 is used to perform job debugging and operation based on the target operator;

[0130] The display module 1103 is used to display the output result of the job debugging operation in real time on the front-end editing interface.

[0131] Optionally, the acquisition module 1101 includes:

[0132] A display submodule, used for displaying the initial job directed acyclic graph on the front-end editing interface;

[0133] The first acquisition submodule is used to acquire the target operator to be debugged and run selected by the user from the initial job directed acyclic graph.

[0134] Optionally, the processing module 1102 includes:

[0135] The startup submodule is used to call the debug mode startup job based on the user's target operation;

[0136] A preprocessing submodule, used for preprocessing the initial job directed acyclic graph based on the target operator to obtain a target job directed acyclic graph;

[0137] The processing submodule is used to debug and run the target job directed acyclic graph.

[0138] Optionally, the preprocessing submodule includes:

[0139] The modification unit is used to modify the initial job directed acyclic graph based on the target operator to obtain the target job directed acyclic graph.

[0140] Optionally, the target job directed acyclic graph is obtained by modifying the initial job directed acyclic graph in any of the following ways:

[0141] Replacing the source operator of the initial job directed acyclic graph with the Websocket source operator; adding a new Websocket sink operator after the target operator of the initial job directed acyclic graph; replacing the sink operator of the initial job directed acyclic graph with the Websocket sink operator;

[0142] Add a Websocket sink operator after the source operator of the initial job directed acyclic graph; add a Websocket source operator before the successor operator of the source operator of the initial job directed acyclic graph; add a new Websocket sink operator after the target operator of the initial job directed acyclic graph; replace the sink operator of the initial job directed acyclic graph with the Websocket sink operator.

[0143] Optionally, the debugging processing device 1100 further includes:

[0144] An estimation submodule, used for re-estimating the resources occupied by the target job according to the number of the Websocket source operators in the directed acyclic graph of the target job and the number of the Websocket sink operators included in the directed acyclic graph of the target job;

[0145] The submitting submodule is used to use the resources occupied by the target job as target parameters for submitting the computing engine job.

[0146] Optionally, the processing submodule includes:

[0147] A submitting unit, configured to submit the target job directed acyclic graph to a computing engine according to the target parameters;

[0148] The processing unit is used to run the directed acyclic graph of the target job through the computing engine to obtain the output result of the job debugging operation.

[0149] An embodiment of the present invention provides a debugging and processing system for real-time operations of computing power in an intelligent computing center, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the real-time operation processing method as described in the above embodiment.

[0150] An embodiment of the present invention provides a machine-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for processing a real-time job as described in the above embodiment are implemented.

[0151] An embodiment of the present invention provides a computer program product, including a computer program, which implements the steps of the debugging processing method of the real-time job as described in the above embodiment when executed by a processor. An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, which implements the various processes of the above-mentioned processing method embodiment of the real-time job when executed by a processor, and can achieve the same technical effect, which will not be repeated here to avoid repetition. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0152] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0154] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A debugging and processing method for real-time operation through intelligent computing center computing power, characterized in that: The method comprises: Step S1: Obtain the target operator selected by the user in the front-end editing interface; Step S2: performing job debugging and operation based on the target operator; Step S3: Displaying the output result of the job debugging operation in real time on the front-end editing interface.

2. The real-time job debugging processing method according to claim 1, characterized in that: The step of obtaining the target operator selected by the user in the front-end editing interface includes: Step S11: displaying the initial job directed acyclic graph on the front-end editing interface; Step S12: Obtain the target operator to be debugged and run selected by the user from the initial job directed acyclic graph.

3. The real-time job debugging processing method according to claim 1 or 2, characterized in that: The debugging and running of the job based on the target operator includes: Step S21: calling the debugging mode to start the job based on the user's target operation; Step S22: preprocessing the initial job directed acyclic graph based on the target operator to obtain a target job directed acyclic graph; Step S23: Debugging and running the target job directed acyclic graph.

4. The real-time job debugging method according to claim 3, characterized in that: Preprocessing the initial job directed acyclic graph based on the target operator to obtain the target job directed acyclic graph includes: Step S221: modifying the initial job directed acyclic graph based on the target operator to obtain the target job directed acyclic graph.

5. The real-time job debugging method according to claim 4, characterized in that: The target job directed acyclic graph is obtained by modifying the initial job directed acyclic graph in any of the following ways: Step S2211: replacing the source operator of the initial job directed acyclic graph with the Websocket source operator; adding a new Websocket sink operator after the target operator of the initial job directed acyclic graph; replacing the sink operator of the initial job directed acyclic graph with the Websocket sink operator; Step S2212: Add a Websocket sink operator after the source operator of the initial job directed acyclic graph; add a Websocket source operator before the successor operator of the source operator of the initial job directed acyclic graph; add a new Websocket sink operator after the target operator of the initial job directed acyclic graph; replace the sink operator of the initial job directed acyclic graph with the Websocket sink operator.

6. The real-time job debugging method according to claim 5, characterized in that: Before debugging and running the target job directed acyclic graph, the method further includes: Step S24: re-estimating the resources occupied by the target job according to the number of the Websocket source operators in the directed acyclic graph of the target job and the number of the Websocket sink operators included in the directed acyclic graph of the target job; Step S25: taking the target job occupied resources as target parameters for submitting the computing engine job.

7. The real-time job debugging method according to claim 6, characterized in that: The debugging and running of the target job directed acyclic graph includes: Step S231: submitting the target job directed acyclic graph to a computing engine according to the target parameters; Step S232: Run the target job directed acyclic graph through the computing engine to obtain an output result of the job debugging operation.

8. A debugging and processing device for real-time computing operations of intelligent computing center computing power, characterized in that: The device comprises: The acquisition module is used to obtain the target operator selected by the user in the front-end editing interface; A processing module, used for debugging and running a job based on the target operator; A display module is used to display the output result of the job debugging operation in real time on the front-end editing interface.

9. A debugging and processing system for real-time computing operations of intelligent computing center computing power, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the real-time job debugging processing method as claimed in any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the real-time job debugging processing method as claimed in any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the real-time job debugging processing method according to any one of items 1 to 7.

Images