Data processing flow detection method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a data processing flow detection method and device, electronic equipment and a medium, and the method comprises the steps: obtaining a configuration file of a target service, configuring a data processing flow of the target service according to the configuration file, simulating the operation of the data processing flow, and carrying out the verification to find a potential problem in a data flow process. And verifying whether the function of the link is correct. And a detection result is displayed in real time through an interactive interface, and exception handling and manual intervention are supported. According to the embodiment of the invention, comprehensive detection and optimization of the data processing flow are realized, the stability and reliability of the data processing flow are improved, and the business risk caused by data processing errors is reduced.

Description

Technical Field

[0001] The present invention relates to the field of big data computing and testing technology, and in particular to a data processing flow detection method and device, electronic equipment and medium. Background Art

[0002] The complexity and scale of data processing processes in modern society are constantly increasing, involving more and more data sources and processing links. Traditional data processing process detection methods usually rely on manual testing and manual verification, which have problems such as low efficiency, prone to errors, and incomplete coverage. With the development of big data and artificial intelligence technology, enterprises have put forward higher requirements for the stability and reliability of data processing processes. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a data processing flow detection method and a corresponding data processing flow detection device, a data processing flow detection electronic device and a medium that overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problems, an embodiment of the present invention discloses a method for detecting a data processing flow, comprising:

[0005] Obtain the configuration file of the target business;

[0006] According to the configuration file, configure the data processing flow of the target business; the data processing flow includes multiple processing links;

[0007] After the data processing flow is configured, a simulation operation is performed on the data processing flow;

[0008] Obtaining processing results of the multiple processing links during the simulation operation;

[0009] The processing results of the multiple processing links are detected.

[0010] Optionally, configuring the data processing flow of the target service according to the configuration file includes:

[0011] Determine, according to the configuration file, a graph structure of a data processing flow of the target business;

[0012] Determining the dependency relationship between the multiple processing links according to the graph structure of the data processing flow of the target business;

[0013] According to the dependency relationship between the multiple processing links, detecting whether the pre-dependencies of the multiple processing links are satisfied;

[0014] When the pre-dependencies of the multiple processing links are all satisfied, the multiple processing links are initialized.

[0015] Optionally, the multiple processing links include a data collection link, a data processing link, a data analysis link, and a data storage link;

[0016] After the data processing flow is configured, the data processing flow is simulated and run, including:

[0017] Acquire simulation data, and input the simulation data into the data entry of the data acquisition link so as to process the simulation data according to the data processing flow.

[0018] Optionally, the detecting the processing results of the multiple processing links includes:

[0019] Obtaining expected results of the multiple processing steps;

[0020] The processing results of the multiple processing links are compared with the corresponding expected results respectively.

[0021] Optionally, it also includes displaying the processing results of the multiple processing links during the simulation operation.

[0022] Optionally, it also includes displaying reminder information when an abnormality is detected in the processing result of the processing link.

[0023] Optionally, it also includes:

[0024] Get the adjustment instruction input by the user;

[0025] According to the adjustment instruction, the configuration parameters of the data processing flow are adjusted.

[0026] Correspondingly, an embodiment of the present invention further discloses a data processing flow detection device, the device comprising:

[0027] Data collection module, used to obtain the configuration files of the target business;

[0028] A front-end configuration module, used to configure the data processing flow of the target business according to the configuration file; the data processing flow includes multiple processing links;

[0029] A simulation operation module, used for performing a simulation operation on the data processing process after the data processing process configuration is completed;

[0030] A result acquisition module, used to obtain the processing results of the multiple processing links in the simulation operation process;

[0031] The result detection module is used to detect the processing results of the multiple processing links.

[0032] Optionally, the pre-configuration module includes:

[0033] A graph structure construction submodule, used to determine the graph structure of the data processing flow of the target business according to the configuration file;

[0034] A dependency determination submodule, used to determine the dependency relationships between the multiple processing links according to the graph structure of the data processing flow of the target business;

[0035] A pre-dependency checking submodule, used to detect whether the pre-dependencies of the multiple processing links are satisfied according to the dependency relationships between the multiple processing links;

[0036] The link initialization submodule is used to initialize the multiple processing links when the pre-dependencies of the multiple processing links are all met.

[0037] Optionally, the multiple processing links include a data acquisition link, a data processing link, a data analysis link, and a data storage link; the simulation operation module includes: a simulation data flow sub-module, used to obtain simulation data, input the simulation data into the data entry of the data acquisition link, so as to process the simulation data according to the data processing flow.

[0038] Optionally, the result detection module includes:

[0039] An expected result acquisition submodule, used to acquire the expected results of the multiple processing links;

[0040] The detection result comparison submodule is used to compare the processing results of the multiple processing links with the corresponding expected results respectively.

[0041] Optionally, the device further comprises:

[0042] A processing result display module is used to display the processing results of the multiple processing links during the simulation operation;

[0043] Optionally, the device further comprises:

[0044] An abnormality reminder module is used to display a reminder message when an abnormality is detected in the processing result of the processing link;

[0045] Optionally, the device further comprises:

[0046] An instruction acquisition module, used to acquire the adjustment instruction input by the user;

[0047] Optionally, the device further comprises:

[0048] The parameter adjustment module is used to adjust the configuration parameters of the data processing flow according to the adjustment instruction.

[0049] Correspondingly, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the various steps of the above-mentioned data processing flow detection method embodiment when executed by the processor.

[0050] Accordingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step of the above-mentioned data processing flow detection method embodiment is implemented.

[0051] The embodiments of the present invention include the following advantages:

[0052] In the embodiment of the present invention, the program first obtains the configuration file of the target business by collecting relevant documents and information of the target business, reads the configuration file to automatically build the graph structure of the data processing flow, and clarifies the dependency relationship between each processing link; then performs pre-dependency verification, link status initialization, data flow simulation and link function verification to ensure the correct function of each link; then displays the detection results to the user in real time, and the user can manually intervene according to the results; finally, according to the adjustment instructions input by the user, adjust the configuration parameters of the data processing flow, and re-execute the detection to ensure the stability and reliability of the data processing flow. Comprehensive detection and optimization of the data processing flow is achieved, the stability and reliability of the data processing flow is improved, and the business risks caused by data processing errors are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a step flow chart of a data processing flow detection method provided by an embodiment of the present invention;

[0054] Figure 2 is a flowchart of another data processing flow detection method provided by an embodiment of the present invention;

[0055] Figure 3 It is a diagram structure schematic diagram of a data processing flow of a data processing flow detection method provided by an embodiment of the present invention;

[0056] Figure 4 It is a program detection logic diagram of a data processing flow detection method provided by an embodiment of the present invention;

[0057] Figure 5 It is a schematic diagram of interactive prompts of a data processing flow detection method provided by an embodiment of the present invention;

[0058] Figure 6It is a structural block diagram of a data processing flow detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] The complexity and scale of data processing processes in modern society are constantly increasing, involving more and more data sources and processing links. Traditional data processing process detection methods usually rely on manual testing and manual verification, which have problems such as low efficiency, error-proneness, and incomplete coverage. With the development of big data and artificial intelligence technologies, enterprises have put forward higher requirements for the stability and reliability of data processing processes. Therefore, an intelligent data processing process detection method is needed that can comprehensively, efficiently and accurately detect each link in the data processing process, ensure the accuracy, integrity and consistency of the data, and improve the stability and reliability of the data processing process.

[0061] One of the core concepts of an embodiment of the present invention is to propose a method for detecting a data processing flow, by obtaining a configuration file of a target business, configuring a data processing flow, and performing a simulation run after the configuration is completed to obtain processing results of multiple processing links, and finally detecting these results to ensure the accuracy, completeness and consistency of the data processing flow, and improve the stability and reliability of the data processing flow.

[0062] Reference Figure 1 , shows a detection logic diagram of a data processing flow detection method provided by an embodiment of the present invention, and the method may specifically include the following steps:

[0063] Step 101, obtaining a configuration file of a target service;

[0064] The target business refers to the specific business or system that needs to be tested for data processing flow. The target business can be various enterprises and scenarios that need to process data. Obtaining the configuration file of the target business means extracting and obtaining the configuration file used to define and describe the data processing flow from the target business system. The configuration file contains detailed information about the data processing flow, such as the definition of each processing link, dependencies, input and output formats, expected results, etc. The configuration file is the basis of the data processing flow, ensuring the correct execution and efficient operation of the data processing flow.

[0065] In the method of the embodiment of the present invention, the program collects relevant documents and materials of the target business, including recording the configuration information required for the operation of the program.

[0066] Step 102, configuring the data processing flow of the target business according to the configuration file; the data processing flow includes multiple processing links;

[0067] In the method of the embodiment of the present invention, the program reads the acquired configuration file of the target business, clarifies the data processing flow of the target business, and determines the detection target, scope and data processing flow link to be detected.

[0068] The data processing flow refers to the entire processing process and steps from data input to data output during the data processing process. The data processing flow usually includes multiple processing links. A link refers to a single processing step or stage in the data processing flow. Each link is responsible for different data processing tasks. Each link is executed in a certain order to ensure the accuracy, integrity and consistency of the data.

[0069] Configuring the data processing flow of the target business according to the configuration file means parsing and building a graph structure of the data processing flow according to the acquired configuration file, clarifying the dependency between various processing links, and configuring various links of the data processing flow.

[0070] Before simulating the data flow, the program will initialize the state of each processing link to ensure that the link is in the correct initial state during the simulation. This step is the basis for subsequent testing and can ensure that the program can accurately understand and simulate the data processing process.

[0071] Step 103, after the data processing process is configured, simulate the data processing process;

[0072] The data processing flow configuration is to parse and build the graph structure of the data processing flow based on the obtained configuration file, clarify the dependencies between various processing links, and configure each link of the data processing flow.

[0073] In one example, after the data processing flow of the target business is configured, the program simulates the flow of data in the data processing flow, tracks the flow path of the simulated data, and simulates the operation of the data processing flow.

[0074] That is, after the configuration file is parsed and the data processing flow is built, the correctness and reliability of the data processing flow are verified by simulating the data flow and processing process.

[0075] The purpose of simulation operation includes verifying whether each link of the data processing process is executed as expected to ensure the correctness of the data processing process; discovering potential problems in the data processing process, such as data loss, format errors, logical errors, etc., to ensure the reliability of the data processing process; and optimizing the performance and efficiency of the data processing process through simulation operation to improve the stability and reliability of the data processing process.

[0076] In one example, the steps of simulating the operation may include:

[0077] Prepare simulation data: Prepare simulation data to simulate the input of the data processing process. The simulation data can be a known test data set to verify each link of the data processing process;

[0078] Simulate data flow: According to the graph structure of the data processing flow, execute each processing link in sequence to simulate the flow and processing of data. By simulating data flow, tracking the flow path of data, and checking whether the changes of data in each processing link are in line with expectations;

[0079] Record processing results: During the simulation operation, record the processing results of each processing link, including data input, output, intermediate processing status and other information. By recording the processing results, you can fully understand the function and performance of each link;

[0080] Verify processing results: Verify the recorded processing results to ensure that the functions and performance of each processing link meet expectations. The verification process includes data format verification, data range verification, data logic verification, etc.

[0081] Step 104, obtaining the processing results of the multiple processing links during the simulation operation;

[0082] In the method of the embodiment of the present invention, the program simulates the data processing flow by tracking the flow path of the simulated data, and records the input, output and intermediate processing status of each link in real time to obtain the processing results of multiple processing links during the simulation operation. By recording the processing results, the function and performance of each link are fully understood to ensure the accuracy, completeness and consistency of the data processing flow.

[0083] Step 105, detecting the processing results of the multiple processing links.

[0084] In one example, after obtaining the processing results of multiple processing links during the simulation operation, the program checks whether the changes in the data in each processing link are in line with expectations, and whether the output of the link is consistent with the expected results, to find potential problems in the data flow process, such as data loss, format errors, etc., and verify whether the functions of the links are correct. This step ensures that each link in the data processing flow can perform its functions as expected.

[0085] In one example, the detection process may include data format verification, data range verification, data logic verification, etc. to ensure the accuracy, completeness and consistency of the data processing flow.

[0086] In some embodiments, the embodiments of the present invention obtain the configuration file of the target business, configure the data processing flow of the target business according to the configuration file, simulate the data processing flow after the configuration is completed, obtain the processing results of multiple processing links in the simulation process, and test these results. Automated detection is achieved, manual operations are reduced, and detection efficiency is improved; by comprehensively covering all links in the data processing process, the accuracy, completeness and consistency of the data are ensured, and the accuracy of detection is improved; by recording the abnormal situation of each processing link and generating corresponding reminder information, it helps users quickly locate the problem link, and improves the speed of problem location; it provides a comprehensive, efficient and accurate detection method, reduces errors in the data processing process, and reduces the business risks caused by data processing errors.

[0087] Reference Figure 2 , shows a program detection logic diagram of another data processing flow detection method of the present invention, which may specifically include the following steps:

[0088] Step 201, obtaining a configuration file of a target service;

[0089] In some embodiments, the program collects documents and materials related to the target business data processing process to obtain all the information required for detection and relevant configuration files of the target business data processing process. All the information required for detection may include storage location information of materials, environmental information, and connection information of components required for the process.

[0090] In some embodiments, the configuration file content may include material information, library table information, task information, component information, and preset received quantity information.

[0091] In some embodiments, the storage location information of the material may include the configuration file path that each link depends on and the data packet path of the simulation data; the environmental information may include the docker version and the python version; the connection information of the components required for the process may be kafka, hive, hbase, ES, and doris.

[0092] Step 202, configuring the data processing flow of the target business according to the configuration file; the data processing flow includes a data collection link, a data processing link, a data analysis link, and a data storage link;

[0093] The data processing flow refers to the entire process and steps of data processing from input to output. The links in the data processing flow usually refer to data collection, cleaning, conversion, analysis and storage, etc., generally including four major links: data collection, data processing, data analysis and data storage;

[0094] In some embodiments, all aspects of the data processing process may include data collection, data processing, data analysis (data extraction, data comparison), and data storage (ods-hive, ods-hbase, dwd-hive, dwd-doris, dim-hive, dim-doris).

[0095] In some embodiments, step 202 may include the following sub-steps:

[0096] Sub-step S11, determining the graph structure of the data processing flow of the target business according to the configuration file;

[0097] In one example, the program collects documents and information related to the data processing flow and obtains the relevant configuration files of the data processing flow of the target business. It also reads the configuration files to clarify the data processing flow of the target business and automatically constructs the graph structure of the data processing flow.

[0098] The graph structure of the data processing flow refers to the use of graphics to represent the relationship and dependency between the various processing links in the data processing flow. The graph structure is usually composed of nodes and edges. The nodes represent the processing links, and the arrows represent the dependency between the processing links. The dependency between the links is indicated by arrows.

[0099] Reference Figure 3 , as shown in a diagram of a data processing flow of a data processing flow detection method provided by an embodiment of the present invention, the diagram structure describes a data processing flow, including multiple processing links and the dependencies between them:

[0100] The data collection link in the figure is the first step of the data processing process, which is responsible for collecting raw data from various data sources; the data processing link depends on the output of the data collection link, and is responsible for pre-processing operations such as cleaning, conversion, and standardization of the collected raw data; the data extraction link depends on the output of the data processing link, and is responsible for extracting useful information or features from the processed data; the ODS link (including ods-hive and ods-hbase) depends on the output of the data extraction link and stores the data in the Hive data warehouse or HBase NoSQL database; the data comparison link depends on the output of the ODS link, and is responsible for comparing data from different data sources or different processing stages to ensure the accuracy and consistency of the data; the DIM link (including dim-hive and dim-doris) depends on the output of the data comparison link and stores the dimensional data in the Hive or Doris data warehouse; the DWD link (including dwd-hive and dwd-doris) depends on the output of the DIM link and stores the detailed data in the Hive or Doris data warehouse.

[0101] Sub-step S12, determining the dependency relationship between the multiple processing links according to the graph structure of the data processing flow of the target business;

[0102] In one example, the graph structure of the data processing flow constructed by the program itself clearly specifies the dependency relationship between each link. Determining the dependency relationship between each processing link according to the graph structure of the data processing flow of the target business may include:

[0103] Analyze graph structure: Analyze the relationship and dependency between each processing link according to the graph structure of the constructed data processing flow;

[0104] Clarify dependencies: Based on the parsed graph structure, clarify the dependencies between various processing links to ensure the correct execution of the data processing flow;

[0105] Configure dependencies: Based on clear dependencies, configure the dependencies between various processing links to ensure the correct execution of the data processing flow;

[0106] Through the above process, the dependency relationship between each processing link is determined according to the graph structure of the data processing flow of the target business to ensure the correct execution of the data processing flow. The target and scope of the test are determined, and the data processing flow link to be tested is determined.

[0107] Sub-step S13, detecting whether the pre-dependencies of the multiple processing links are satisfied according to the dependency relationships between the multiple processing links;

[0108] In one example, before the formal test, the program checks whether the pre-dependencies of each processing link are met. Pre-dependencies refer to dependency conditions that must be met before a processing link is executed. Pre-dependencies are usually the outputs of other processing links to ensure the correct execution of the data processing flow. Including configuration information related to the current link, the library tables or kafka topics required to store data, and the inspection results of the parent dependency modules. Specifically, it verifies and checks the environmental materials, library table status, and task status to check whether the configuration is complete, that is, the program verifies whether the configuration parameters are valid. Configuration parameters refer to the parameters defined in the configuration file for configuring the data processing flow, such as the definition of the processing link, dependencies, input and output formats, expected results, etc.

[0109] Sub-step S14, initializing the multiple processing links when the pre-dependencies of the multiple processing links are all satisfied.

[0110] Initialization refers to the initial setting of various components, variables, objects, etc. when the program or system starts to ensure that they are in the correct initial state during operation. The purpose of initialization is to ensure that the program or system can run normally and avoid errors and exceptions caused by uninitialized or incorrect initialization.

[0111] In one example, before the data flow simulation, when the pre-dependencies of multiple processing links are met, the program will initialize the state of each processing link to ensure that the link is in the correct initial state during the simulation. This step is the basis for link function verification and ensures the accuracy of the verification results.

[0112] In some embodiments, the program initializes the state of each processing link, including configuring the detection environment, configuring the software and tools that each processing link depends on, and configuring the detection parameters, such as setting the initial parameters of the link, recording the repository metadata, etc.

[0113] Step 203, after the data processing process is configured, simulate the data processing process;

[0114] In some embodiments, the step 203 may include: acquiring simulation data, inputting the simulation data into the data entry of the data acquisition link, and processing the simulation data according to the data processing flow.

[0115] Simulation data refers to the data used to simulate the input of the data processing process during the simulation operation of the data processing process. Simulation data can be a known test data set used to verify each link of the data processing process.

[0116] In one example, after the data processing flow is configured, the program simulates the flow of data in the data processing flow and performs a simulated operation of the data processing flow.

[0117] Reference Figure 4 As shown in the program detection logic diagram of a data processing flow detection method provided by an embodiment of the present invention, the pre-prepared data packet is designated to a storage location in the "Material Configuration". When the step "Inputting Original Data" is executed, the program parses the data packet to obtain simulated data and puts the data into the warehouse from "Data Collection". By inputting simulated data into the "Data Collection" entrance, each processing link is executed in sequence according to the graph structure of the data processing flow, simulating the flow and processing process of data, and the program will track the flow path of data.

[0118] Step 204, obtaining the processing results of the multiple processing links during the simulation operation;

[0119] In one example, by inputting simulation data into the "Data Collection" entry, the program will track the flow path of the data and record the input, output, and intermediate processing status of each link in real time. During the simulation run, the program will record the processing results of each processing link, which include information such as data input, output, and intermediate processing status. By recording the processing results, the program can fully understand the function and performance of each link. Check whether the changes in data in each processing link are in line with expectations. This step can discover potential problems in the data flow process, such as data loss, format errors, etc.

[0120] At the same time, the program will perform functional verification for each link in the data processing flow. By inputting a known simulated data set into the link, which can be the simulated data input into the "Data Collection" entrance, and checking whether the output of the link is consistent with the expected result, the program can verify whether the function of the link is correct. This step ensures that each link in the data processing flow can perform its function as expected.

[0121] Step 205: detecting the processing results of the multiple processing links.

[0122] In some embodiments, step 205 may include the following sub-steps:

[0123] Sub-step S21, obtaining expected results of the multiple processing steps;

[0124] In some embodiments, when the initial state of the environment and the data processing strategy remain unchanged, the state of the same batch of data flowing to the same link multiple times is the same. Before the process detection, the state of the data in each link is detected and recorded in a complete environment using predetermined simulation data, and then the state information is recorded in the configuration file to determine the flow process of the data in the data processing process and the function of each link.

[0125] In one example, during the simulation of the data processing flow, the program will obtain the expected results of each processing link from the configuration file or repository as the standard for detection. The reading process can be real-time reading or batch reading, which is selected according to actual needs.

[0126] In another example, the program can load the expected results of the reading into the data flow simulation engine as the verification criteria for each processing link.

[0127] Sub-step S22, respectively comparing the processing results of the multiple processing links with the corresponding expected results.

[0128] In one example, the program compares the processing results with the corresponding expected results at each processing step. The comparison process may include data format verification, data range verification, data logic verification, etc. The program records the comparison results of each processing step.

[0129] The record of comparison results can help users quickly identify problematic links. The comparison results of each processing link are stored in a designated repository, such as a database, file system, etc.

[0130] In some embodiments, the selection and configuration of the storage library can be performed according to actual needs to ensure the storage and access efficiency of the comparison results.

[0131] Step 206: during the simulation operation, display the processing results of the multiple processing links.

[0132] In one example, the program will display the comparison results of each processing link to the user in an intuitive way. The display method can be in the form of charts, tables, logs, etc., which is easy for users to understand and analyze. Users can view the comparison results of each processing link through an interactive interface.

[0133] Reference Figure 5 , a schematic diagram of interactive prompts of a data processing flow detection method provided by an embodiment of the present invention.

[0134] The interactive prompt diagram shows a menu of system detection functions, through which the user can select different modules for detection to ensure the normal operation of the system at each stage. The menu options include full-process detection, which executes all detection steps, covering a comprehensive check from environment configuration to service status, ensuring the integrity and correctness of the entire process; environment and material detection, which is used to check the environment configuration and material status required for system operation to ensure that all necessary resources and conditions are ready; data status initialization, which is used to initialize the data status in the system to ensure that the data is in the correct initial state before processing begins; raw data input, which puts the original data into the system to check whether the data is correctly loaded and ready for subsequent processing; library table status detection, including TOPIC and HIVE tables of KAFKA cluster 1, respectively detects the TOPIC status and HIVE table status in the KAFKA cluster to ensure the normal function of message queues and data storage query functions; service status detection, including access service and preprocessing service, respectively detects the access service status and running status of the preprocessing service of the system to ensure that external requests can be correctly connected to the system and that data is correctly preprocessed before entering the core processing flow; and wild inspection status, which performs some additional, non-standardized inspection tasks to ensure the stability and reliability of the system in various edge cases. Each option corresponds to a specific detection task. Through these detection steps, the system administrator can fully understand the operating status of the system, discover and solve potential problems in a timely manner, and ensure the normal operation of the system at different stages.

[0135] Step 207: When an abnormality is detected in the processing result of the processing link, a reminder message is displayed.

[0136] During the simulation, the program verifies the processing results of each processing link to detect whether there are any abnormal conditions. In one example, the abnormal conditions may include data format errors, data range exceeded, data logic errors, etc.

[0137] In one example, the program records the abnormal situation of each processing link, including the abnormal type, abnormal location, abnormal data, etc. Recording the abnormality can help users quickly identify the problem link.

[0138] When an abnormality is detected in the processing result of the processing link, the program will generate corresponding reminder information. In one example, the reminder information may include the abnormal description, abnormal cause, abnormal location, abnormal data, etc. The program will display the generated reminder information to the user in real time. In one example, the display method can be in the form of pop-up windows, log output, charts, etc., so that users can understand the abnormal situation in a timely manner. Users can view reminder information through an interactive interface.

[0139] Step 208, obtaining an adjustment instruction input by the user;

[0140] In one example, during the simulation operation, the program provides an interactive interface through which the user can input adjustment instructions; the program parses the adjustment instructions input by the user to identify the type and specific content of the adjustment instructions.

[0141] In some embodiments, the adjustment instruction input by the user may include modifying configuration parameters, adjusting detection strategies, viewing detailed logs, locating problems, etc. The type of adjustment instruction is selected according to actual needs.

[0142] During the simulation of the data processing flow, users can input adjustment instructions through the interactive interface to adjust the configuration parameters of the data processing flow to optimize the performance and efficiency of the data processing flow. By obtaining the adjustment instructions input by the user, the user can adjust the configuration parameters of the data processing flow to optimize the performance and efficiency of the data processing flow and improve the stability and reliability of the data processing flow. Users can also flexibly adjust the configuration parameters of the data processing flow according to the test results and reminder information to further improve the user experience.

[0143] Step 209: adjusting the configuration parameters of the data processing flow according to the adjustment instruction.

[0144] In one example, during the detection process, the program will display the detection results and reminder information to the user in real time. The user can view the detection results and reminder information through an interactive interface and input adjustment instructions as needed.

[0145] The program will parse the adjustment instruction input by the user, identify the type and specific content of the adjustment instruction, extract the specific content of the adjustment instruction according to the identified adjustment instruction type, verify the validity of the adjustment instruction, and ensure the correctness and integrity of the adjustment instruction. In some embodiments, the verification process includes data format verification, data range verification, data logic verification, etc. According to the parsed adjustment instruction, the program will perform corresponding operations to adjust the configuration parameters of the data processing flow.

[0146] In one example, after adjusting the configuration parameters of the data processing flow, the program will re-execute the test to verify whether the adjusted data processing flow meets expectations. Re-execution of the test can help users confirm the effect of the adjustment and ensure the stability and reliability of the data processing flow.

[0147] In some embodiments, the program will record the results of each adjustment, including configuration parameters before and after the adjustment, adjustment instructions, adjustment effects, etc. Recording the adjustment results can help users analyze and summarize the adjustment effects and optimize the data processing process.

[0148] The embodiment of the present invention realizes automated detection, real-time feedback and interactive adjustment through an intelligent data processing flow detection method, comprehensively covers all links in the data processing flow, ensures the accuracy, completeness and consistency of the data, and monitors the data processing process through intelligent tasks, so as to quickly discover problems and locate them, convert manual testing methods into intelligent tasks, improve detection efficiency, improve the stability and reliability of the data processing flow, reduce business risks, and enhance user experience.

[0149] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0150] Reference Figure 6 , shows a structural block diagram of a data processing flow detection device provided by an embodiment of the present invention, which may specifically include the following modules:

[0151] The data collection module 301 is used to obtain the configuration file of the target business;

[0152] A pre-configuration module 302 is used to configure the data processing flow of the target service according to the configuration file; the data processing flow includes multiple processing links;

[0153] A simulation operation module 303 is used to simulate the data processing process after the data processing process configuration is completed;

[0154] The result acquisition module 304 is used to obtain the processing results of the multiple processing links in the simulation operation process;

[0155] The result detection module 305 is used to detect the processing results of the multiple processing links.

[0156] In an embodiment of the present invention, the pre-configuration module includes:

[0157] A graph structure construction submodule, used to determine the graph structure of the data processing flow of the target business according to the configuration file;

[0158] A dependency determination submodule, used to determine the dependency relationships between the multiple processing links according to the graph structure of the data processing flow of the target business;

[0159] A pre-dependency checking submodule, used to detect whether the pre-dependencies of the multiple processing links are satisfied according to the dependency relationships between the multiple processing links;

[0160] The link initialization submodule is used to initialize the multiple processing links when the pre-dependencies of the multiple processing links are all met.

[0161] In an embodiment of the present invention, the multiple processing links include a data acquisition link, a data processing link, a data analysis link, and a data storage link; the simulation operation module includes: a simulation data flow sub-module, which is used to obtain simulation data and input the simulation data into the data entry of the data acquisition link to process the simulation data according to the data processing flow.

[0162] In an embodiment of the present invention, the result detection module includes:

[0163] An expected result acquisition submodule, used to acquire the expected results of the multiple processing links;

[0164] The detection result comparison submodule is used to compare the processing results of the multiple processing links with the corresponding expected results respectively.

[0165] In an embodiment of the present invention, the device further includes:

[0166] A processing result display module is used to display the processing results of the multiple processing links during the simulation operation;

[0167] In an embodiment of the present invention, the device further includes:

[0168] An abnormality reminder module is used to display a reminder message when an abnormality is detected in the processing result of the processing link;

[0169] In an embodiment of the present invention, the device further includes:

[0170] An instruction acquisition module, used to acquire the adjustment instruction input by the user;

[0171] In an embodiment of the present invention, the device further includes:

[0172] The parameter adjustment module is used to adjust the configuration parameters of the data processing flow according to the adjustment instruction.

[0173] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0174] An embodiment of the present invention further provides an electronic device, including:

[0175] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data processing flow detection method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0176] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned data processing flow detection method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0177] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0178] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0179] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0182] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0183] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0184] The data processing flow detection method and device provided by the present invention are introduced in detail above. The principles and implementation methods of the present invention are explained in this article using specific examples. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting a data processing flow, characterized in that: include: Obtain the configuration file of the target business; According to the configuration file, configure the data processing flow of the target business; The data processing flow includes multiple processing steps; After the data processing flow is configured, a simulation operation is performed on the data processing flow; Obtaining processing results of the multiple processing links during the simulation operation; The processing results of the multiple processing links are detected.

2. The method according to claim 1, characterized in that The step of configuring the data processing flow of the target service according to the configuration file includes: Determine, according to the configuration file, a graph structure of a data processing flow of the target business; Determining the dependency relationship between the multiple processing links according to the graph structure of the data processing flow of the target business; According to the dependency relationship between the multiple processing links, detecting whether the pre-dependencies of the multiple processing links are satisfied; When the pre-dependencies of the multiple processing links are all satisfied, the multiple processing links are initialized.

3. The method according to claim 1, characterized in that The multiple processing links include data collection link, data processing link, data analysis link, and data storage link; After the data processing flow is configured, the data processing flow is simulated and run, including: Acquire simulation data, and input the simulation data into the data entry of the data acquisition link so as to process the simulation data according to the data processing flow.

4. The method according to claim 1, characterized in that The detecting the processing results of the multiple processing links includes: Obtaining expected results of the multiple processing steps; The processing results of the multiple processing links are compared with the corresponding expected results respectively.

5. The method according to claim 1, characterized in that Also includes: During the simulation operation, the processing results of the multiple processing links are displayed.

6. The method according to claim 1, characterized in that Also includes: When an abnormality is detected in the processing result of the processing link, a reminder message is displayed.

7. The method according to claim 1, characterized in that Also includes: Get the adjustment instruction input by the user; According to the adjustment instruction, the configuration parameters of the data processing flow are adjusted.

8. A data processing flow detection device, characterized in that: The device comprises: Data collection module, used to obtain the configuration files of the target business; A front-end configuration module, used to configure the data processing flow of the target business according to the configuration file; the data processing flow includes multiple processing links; A simulation operation module, used for performing a simulation operation on the data processing process after the data processing process configuration is completed; A result acquisition module, used to obtain the processing results of the multiple processing links in the simulation operation process; The result detection module is used to detect the processing results of the multiple processing links.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the data processing flow detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing flow detection method according to any one of claims 1 to 7 are implemented.

Images