Data management platform, data management method and device, medium and equipment

Abstract

The invention discloses a data management platform, a data management method, a data management device, a medium and equipment. The data management platform can store an intermediate execution result processed by each data processing layer into a second storage module in a process of executing a target code through a first data link of an execution module, so that when a backtracking operation needs to be executed for the target code, the backtracking operation can be executed for the target code. According to the data management platform, the intermediate execution result processed by each data processing layer of the first data link is obtained from the second storage module through the independent data link to perform the backtracking operation, so that the influence of the execution of the backtracking operation on the execution of the real-time data processing operation can be avoided, and the data backtracking capability of the data management platform is improved.

Description

Technical Field

[0001] The present invention relates to the field of data engineering technology, and in particular to a data management platform and a data management method, device, medium and equipment. Background Art

[0002] The data management platform includes functions such as data storage, data cleaning, data conversion, data calculation, and data analysis. Therefore, the data management platform is widely used in many fields. For example, in the field of product development, users can use the data management platform to analyze consumer behavior, market trends, etc. to help users formulate more accurate market strategies. Another example: in the field of risk management, users can use the data management platform to analyze historical data and real-time data to identify potential risks, such as credit risk assessment, supply chain risk warning, etc., to help users take preventive measures.

[0003] Normally, when a data management platform processes data, it can treat all data processing as stream processing. Even the processing of historical data is completed by replaying the data stream through the stream processing engine, thereby realizing the unification of stream processing and batch processing, simplifying the system complexity and reducing maintenance costs. Among them, stream processing means that the data processing task is executed immediately once it is generated, which can improve the real-time and continuity of the data processing task. It is usually used to process high-frequency and continuously generated data processing tasks. Batch processing means collecting and storing each data processing task within a specific time period as a batch, and then uniformly processing all data processing tasks in a batch. It is usually used to process historical data or static data sets.

[0004] Although the data management platform can treat all data processing as stream processing, thereby simplifying the complexity of the system, when data backtracking is required (that is, when historical data needs to be processed or large quantities of data need to be recalculated), the demand for computing resources will increase in a short period of time, which will have a great impact on the operation of the computing engine of the data management platform.

[0005] Therefore, how to improve the data backtracking capability of the data management platform is an urgent problem to be solved. Summary of the invention

[0006] This specification provides a data management platform and a data management method, device, medium and equipment to partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions:

[0008] This specification provides a data management platform, the data management platform comprising: an execution module, a first storage module, and a second storage module, the execution module is provided with a first data link and a second data link, the first data link and the second data link are mainly composed of various data processing layers;

[0009] The data management platform receives the target code through the execution module, and runs each data processing layer in the order between the data processing layers in the first data link to obtain the execution result corresponding to the target code, wherein for each data processing layer, the data processing layer is run so that the data processing layer sends a data acquisition request to the first storage module to obtain the data to be processed corresponding to the data processing layer, and sends the data processed by the data processing layer as an intermediate execution result to the first storage module for storage as the data to be processed corresponding to the next data processing layer, and sends the intermediate execution result to the second storage module for storage; and

[0010] When it is determined that a backtracking operation needs to be performed on the target code, a historical data acquisition request is sent to the second storage module through each data processing layer in the second data link to obtain the historical data required for executing the target code from the second storage module, and process the historical data to obtain the processed historical data, which is the intermediate execution result generated by the target code during the execution process.

[0011] Optionally, the data management platform further includes: a service module;

[0012] The data management platform generates configuration instructions through the service module according to the configuration operations performed by the user in the preset task interface, and determines the original code input by the user according to the configuration instructions, converts the original code into a specified format, obtains the target code, and sends the target code to the execution module.

[0013] Optionally, the data management platform determines the original code input by the user and the data processing strategy corresponding to the original code according to the configuration instruction through the service module;

[0014] If it is determined that the data processing strategy corresponding to the original code is the first data processing strategy, the original code is converted according to the code format corresponding to the first data processing strategy to obtain the converted original code as the target code, and the target code is sent to the execution module, and the first data processing strategy is used to perform stream processing on each piece of data that is continuously generated;

[0015] The execution model is used to receive the target code, and execute the data processing task corresponding to the target code through the data processing layers in the first data link to obtain the execution result corresponding to the target code.

[0016] Optionally, the data management platform determines the original code input by the user and the data processing strategy corresponding to the original code according to the configuration instruction through the service module;

[0017] If it is determined that the data processing strategy corresponding to the original code is the second data processing strategy, the original code is converted according to the code format corresponding to the second data processing strategy to obtain the converted original code as a batch code, and the batch code is sent to the execution module, the second data processing strategy is used to batch process the data in the specified data set, and the amount of data contained in the specified data set remains unchanged;

[0018] The execution model is used to receive the batch processing code, and execute the data processing task corresponding to the batch processing code through the data processing layers in the second data link to obtain the execution result corresponding to the batch processing code.

[0019] Optionally, the data management platform further includes: a scheduling module;

[0020] When the data management platform determines through the service module that the data processing strategy corresponding to the original code is the second data processing strategy, the batch processing code is sent to the scheduling module;

[0021] When the scheduling module determines that the received batch processing code meets the preset batch processing condition, the received batch processing code is sent to the execution module.

[0022] Optionally, the data management platform further includes: an analysis module;

[0023] The data management platform sends the execution result corresponding to the target code to the analysis module through the execution module; and

[0024] Sending the execution result corresponding to the batch code to the analysis module;

[0025] The analysis module receives the execution result corresponding to the target code and the execution result corresponding to the batch code, and reorganizes the execution result corresponding to the target code and the execution result corresponding to the batch code according to preset dimensions to obtain the reorganized execution result, wherein the dimensions include: time, geographic location, data category, and user group.

[0026] Optionally, the data management platform further includes: an anomaly detection module;

[0027] The data management platform determines at least one data processing layer from the data processing layers run by the execution module when executing the target code through the anomaly detection module, as the target data processing layer, obtains the data to be processed corresponding to the target data processing layer, and detects the data to be processed corresponding to the target data processing layer according to preset anomaly detection rules to obtain the detection result corresponding to the target data processing layer, and the anomaly detection rule is determined in advance based on the historical data corresponding to the execution module.

[0028] Optionally, the first storage module and the second storage module use the same logical storage layer.

[0029] This specification provides a data management method, which is applied to an execution module of a data management platform, wherein the data management platform includes: an execution module, a first storage module, and a second storage module, wherein a first data link and a second data link are provided in the execution module, wherein the first data link and the second data link are mainly composed of data processing layers, and wherein the method includes:

[0030] receiving object code;

[0031] Running each data processing layer in the first data link in the order between the data processing layers to obtain the execution result corresponding to the target code, wherein for each data processing layer, running the data processing layer so that the data processing layer sends a data acquisition request to the first storage module to obtain the data to be processed corresponding to the data processing layer, and sends the data processed by the data processing layer as an intermediate execution result to the first storage module for storage as the data to be processed corresponding to the next data processing layer, and sends the intermediate execution result to the second storage module for storage; and

[0032] When it is determined that a backtracking operation needs to be performed on the target code, a historical data acquisition request is sent to the second storage module through the data processing layers in the second data link to obtain the historical data required for executing the target code from the second storage module, and process the historical data to obtain the processed historical data, which is the intermediate execution result generated during the execution of the target code.

[0033] This specification provides a data management device, the device comprising:

[0034] A receiving module, used for receiving target code;

[0035] a first execution module, configured to run each data processing layer in the first data link in the order between the data processing layers to obtain an execution result corresponding to the target code, wherein for each data processing layer, the data processing layer is run so that the data processing layer sends a data acquisition request to the first storage module to obtain the data to be processed corresponding to the data processing layer, and sends the data processed by the data processing layer as an intermediate execution result to the first storage module for storage as the data to be processed corresponding to the next data processing layer, and sends the intermediate execution result to the second storage module for storage; and

[0036] The second execution module is used to send a historical data acquisition request to the second storage module through the data processing layers in the second data link when it is determined that a backtracking operation needs to be performed on the target code, so as to obtain the historical data required for executing the target code from the second storage module, and process the historical data to obtain the processed historical data, wherein the historical data is an intermediate execution result generated during the execution of the target code.

[0037] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data management method is implemented.

[0038] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data management method when executing the program.

[0039] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0040] In the data management platform provided in the present specification, the data management platform receives the target code through the execution module, and runs each data processing layer in the order between the data processing layers in the first data link to obtain the execution result corresponding to the target code, wherein for each data processing layer, the data processing layer is run so that the data processing layer sends a data acquisition request to the first storage module to obtain the data to be processed corresponding to the data processing layer, and sends the data processed by the data processing layer as an intermediate execution result to the first storage module for storage, as the data to be processed corresponding to the next layer of data processing layer, and sends the intermediate execution result to the second storage module for storage, and, when it is determined that a backtracking operation needs to be performed on the target code, a historical data acquisition request is sent to the second storage module through each data processing layer in the second data link to obtain the historical data required for processing to execute the target code from the second storage module, and the historical data is processed to obtain the processed historical data, where the historical data is the intermediate execution result generated during the execution of the target code.

[0041] It can be seen from the above method that the data management platform can use the first storage module as a message middleware to facilitate the execution module to read and write data during the execution of the target code, and can use the second storage module as a data lake for storing batches of historical data, so that the intermediate execution results processed by each data processing layer can be saved in the second storage module through the first data link of the execution module during the execution of the target code, so that when it is necessary to perform a backtracking operation on the target code, the intermediate execution results processed by each data processing layer of the first data link can be obtained from the second storage module through a separate data link to perform a backtracking operation, thereby avoiding the impact of the backtracking operation on the execution of real-time data processing operations, so as to improve the data backtracking capability of the data management platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0043] Figure 1 A schematic diagram of a data management platform provided in this specification;

[0044] Figure 2 A schematic diagram of the object code execution process provided in this specification;

[0045] Figure 3 This is a schematic diagram of the preset task interface;

[0046] Figure 4A schematic diagram of an analysis module provided in this specification;

[0047] Figure 5 A schematic diagram of the anomaly detection process provided in this specification;

[0048] Figure 6 A flowchart of a data management method provided in this specification;

[0049] Figure 7 A schematic diagram of a data management device provided in this specification;

[0050] Figure 8 A method corresponding to the Figure 6 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0052] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0053] Figure 1 A schematic diagram of a data management platform provided in this specification.

[0054] Combination Figure 1 It can be seen that the above-mentioned data management platform includes: an execution module, a first storage module, and a second storage module.

[0055] The execution module may be provided with components for executing stream processing tasks and batch processing tasks, such as Apache Flink (Flink).

[0056] The first storage module may be provided with a component for storing real-time data streams, such as Kafka (Apache Kafka, Kafka).

[0057] The second storage module may be provided with a component for storing a static data set (the amount of data included in the static data set remains unchanged during the processing), for example: Hudi (Hadoop Upserts and Deletes Incrementally, Hudi).

[0058] In the above content, the data management platform can receive the target code through the execution module, and run each data processing layer in the order between the data processing layers in the first data link to obtain the execution result corresponding to the target code, and run each data processing layer to make the data processing layer send a data acquisition request to the first storage module to obtain the data to be processed corresponding to the data processing layer, and send the data processed by the data processing layer as an intermediate execution result to the first storage module for storage as the data to be processed corresponding to the next data processing layer, and send the intermediate execution result to the second storage module for storage.

[0059] In the above content, the target code may be a collection of a series of instructions sent by the user for controlling the data management platform to perform tasks of processing and analyzing data.

[0060] In the above content, different data processing layers are used to perform different data operations. The above data processing layers may include: operational data storage layer (Operational Data Store, ODS), detailed data layer (Data Warehouse Detail, DWD), data warehouse summary layer (Data Warehouse Summary, DWS), etc.

[0061] In the above content, the ODS layer can be used to obtain and store the original data required to execute the target code. For example, if an e-commerce company's order system generates a large amount of order data every day, when it is necessary to execute the code for analyzing and processing the order data, the order data can be obtained through the ODS layer as the original data, wherein the method for the ODS layer to obtain the order data can be to obtain the order data from the first storage module, or the ODS layer can obtain the order data from a specified database (for example: Mysql and other relational databases).

[0062] In the above content, the DWD layer can be used to further clean, convert, and process the original data. For example, the DWD layer can clean the original data obtained by the ODS layer to filter out invalid data, null values, and abnormal values ​​in the original data, and convert the data type of each original data (for example, the data type of some original data may be floating point data. In this case, in order to facilitate calculation, the data type of this part of the original data can be converted to integer).

[0063] In the above content, the DWS layer can be used to aggregate, summarize, and calculate the cleaned data output by the DWD layer. For example, the DWS layer can aggregate the cleaned data output by the DWD layer according to different time dimensions, such as aggregating the order data in the order system of an e-commerce company according to days, weeks, and months, and obtain the aggregated results as the execution results corresponding to the target code.

[0064] For ease of understanding, the following will describe in detail the process of the data management platform executing the target code through the execution module. Figure 2 shown.

[0065] Figure 2 This is a schematic diagram of the target code execution process provided in this specification.

[0066] Combination Figure 2 It can be seen that after the execution module receives the target code, it can execute the data processing task corresponding to the target code through the data processing layers contained in the first data link, wherein the ODS layer in the first data link can obtain the original data required to execute the data processing task corresponding to the target code from the first storage module or the designated database, and then save the obtained original data in the first storage module and in the second storage module.

[0067] It is worth noting that, since the execution module has temporarily stored the acquired original data or the data processed by each data processing layer in the first storage module when executing the data processing tasks corresponding to each historical code, at this time, when the execution module executes the target code, it can first obtain the original data required for the data processing task corresponding to the target code from the first storage module through the ODS layer. If the original data required for the data processing task corresponding to the target code does not exist in the first storage module, it can be obtained from the designated database. This can improve the data reuse rate, improve the efficiency of obtaining the original data required for the data processing task corresponding to the target code, and avoid the situation where the performance of the designated database is affected due to the execution module frequently obtaining data from the designated database.

[0068] For example: if there is a data processing task corresponding to a historical code that is to aggregate all transaction data of a company in a day by hour, when the execution module executes the data processing task corresponding to the above historical code, it has already obtained all transaction data of a company in this day from the designated database and saved it in the first storage module. When the data processing task corresponding to the target code is to determine the total transaction amount of all transaction data of a company in one day, when the execution module executes the data processing task corresponding to the above target code, it can directly reuse all transaction data of a company in this day that has been stored in the first storage module without accessing the designated database again.

[0069] Furthermore, the DWD layer in the first data link can send a data acquisition request to the first storage module, so that the first storage module responds to the data acquisition request sent by the DWD layer in the first data link, and returns the original data saved by the ODS layer in the first storage module to the DWD layer as the processed data corresponding to the DWD layer, and then the DWD layer can perform data cleaning and other operations on the processed data corresponding to the DWD layer to obtain the cleaned data, and save the cleaned data in the first storage module and in the second storage module.

[0070] Furthermore, the DWS layer in the first data link can send a data acquisition request to the first storage module, so that the first storage module responds to the data acquisition request sent by the DWS layer and returns the cleaned data saved by the DWD layer in the first storage module to the DWS layer as the processed data corresponding to the DWS layer, and then the DWS layer can perform operations such as data aggregation on the processed data corresponding to the DWS layer to obtain the execution result corresponding to the target code.

[0071] From the above content, it can be seen that in the process of executing the target code through the execution module, the data management platform can use the first storage module as the message middleware (which can be understood as using the first storage module as a cache), thereby reducing the delay in the execution module obtaining data in the process of executing the target code, and thus facilitating the execution module to process the data.

[0072] It should be noted that, for each data processing layer of the execution module, after the data to be processed is processed by the data processing layer, a method for sending the data processed by the data processing layer to the first storage module and the second storage module for storage can be to send the data processed by the data processing layer to the first storage module and the second storage module for storage through a change data capture (Flink Change Data Capture, Flink CDC) component preset in the execution module.

[0073] In actual application scenarios, when the data management platform executes the target code through the execution module, there may be reasons such as: data quality problems in the original data obtained, the need for business analysts to re-analyze the original data, caliber adjustment (that is, adjustment of the dimensional indicators and calculation rules used when aggregating the processed data), and errors in the output execution results, which may lead to the need to backtrack the target code.

[0074] At this time, when the data management platform determines that a backtracking operation needs to be performed on the target code, it can send a historical data acquisition request to the second storage module through each data processing layer in the second data link of the execution module to obtain the historical data required for executing the target code from the second storage module, and process the historical data to obtain the processed historical data. The historical data here is the intermediate execution result generated during the execution of the target code.

[0075] The historical data required for executing the target code obtained from the second storage module can be determined according to the backtracking operation actually performed.

[0076] For example: when it is determined that the acquired original data has data quality problems that require a backtracking operation on the target code, the acquired original data needs to be cleaned again. At this time, the data management platform can determine the intermediate execution result output by the ODS layer as the historical data required for processing to execute the target code obtained from the second storage module, and then reprocess the historical data by running the DWD layer and DWS layer of the second data link to obtain the processed historical data as the backtracking execution result corresponding to the target code.

[0077] Another example: when it is determined that a caliber adjustment is required, resulting in the need to perform a backtracking operation on the target code, since only the cleaned data needs to be re-aggregated and other operations, at this time, the data management platform can determine the intermediate execution result output by the DWD layer as the historical data required for executing the target code obtained from the second storage module, and then reprocess the historical data by running the DWS layer of the second data link to obtain the processed historical data as the backtracking execution result corresponding to the target code.

[0078] It should be noted that when each data processing layer in the second data link processes the historical data, the intermediate execution results generated can be saved in the second storage module.

[0079] It can be seen from the above content that the data management platform can use the first storage module as a message middleware to facilitate the execution module to read and write data during the execution of the target code, and can use the second storage module as a data lake for storing batches of historical data. Therefore, during the execution of the target code, the intermediate execution results processed by each data processing layer can be saved in the second storage module through the first data link of the execution module. When a backtracking operation is required for the target code, the intermediate execution results processed by each data processing layer of the first data link can be obtained from the second storage module through a separate data link to perform a backtracking operation, thereby avoiding the impact of the backtracking operation on the execution of real-time data processing operations, thereby improving the data backtracking capability of the data management platform.

[0080] It should be noted that the first storage module and the second storage module mentioned above use the same logical storage layer. The logical storage layer here is used to organize, manage and operate data at the software level, rather than the actual storage layout of the data on the physical medium. Among them, building a logical storage layer can facilitate users or applications to understand and use data, hiding the specific details of the underlying physical storage. It can be understood that the above-mentioned logical storage layer is the display form when displaying data to users.

[0081] For example, in the above-mentioned logical storage layer, data can be organized into a form of presentation such as files, tables, key-value pairs, and document logical structures and presented to users. At this time, in the user's cognition, when operating on data, the operation is performed on the data in the above-mentioned presentation form, such as operating on the data in the file, operating on the data in the table, etc.

[0082] Furthermore, when the logical storage layer used by the first storage module and the second storage module is the same, the user can write the same code to operate the data in the first storage module and / or the second storage module, without having to write the code once when operating the data in the first storage module and write the code again when operating the data in the second storage module, thereby improving the efficiency of the user's operations on the data stored in the first storage module and the second storage module.

[0083] In addition, since the first storage module and the second storage module use the same logical storage layer, the logical structure for storing data set in the logical storage layer can be reused after creation when executing various codes that operate on the data stored in the first storage module and / or the second storage module, thereby avoiding setting different logical structures for different storage modules, thereby reducing the storage resources required by the data management platform.

[0084] In addition, since the first storage module and the second storage module use the same logical storage layer, the method for managing the metadata of the logical storage layer used by the first storage module and the second storage module is also unified, thereby reducing the workload of managing the metadata of the logical storage layer.

[0085] The metadata of the above-mentioned logical storage layer may refer to information that describes the data structure, organization, and management rules in the logical storage layer, such as: the name of the table used to store data, the row information contained in the table used to store data, the column information contained in the table used to store data, the constraint relationship between multiple tables used to store data (which can be understood as the association relationship between different tables), the access rights of the table used to store data, the index settings of the table used to store data, etc.

[0086] It is worth noting that the above-mentioned data management platform may also be provided with a metadata management module, wherein the metadata management module may be provided with components for managing the logical storage layer, such as: meta storage (Hive MetaStore, MetaStore), cataloging (Hive Catalog, Catalog), where the Hive MetaStore is used to receive user operation instructions for managing the metadata of the logical storage layer, and configure the metadata of the logical storage layer according to the received operation instructions, and the Hive Catalog is used to persist the metadata of the configured logical storage layer, so that when the code sent by the user is subsequently received and executed, the metadata of the logical storage layer can be reused without having to reconfigure the metadata of the logical storage layer.

[0087] It should be noted that when storing various historical data in the above-mentioned second storage module, at least part of the historical data can be merged into one file for storage, thereby improving the efficiency of managing historical data. For example: if the historical data of the past month is frequently read and written, the second storage module can divide all historical data in the past month into a partition and save it into a file, so that when reading and writing this part of the historical data subsequently, only the file needs to be read and written.

[0088] In this specification, the data management platform may also be provided with a service module, and the data management platform may display a preset task interface to the user through the service module, and then generate configuration instructions according to the configuration operations performed by the user in the preset task interface, and determine the original code input by the user according to the configuration instructions, and convert the original code into a specified format to obtain a target code, and send the target code to the execution module, as shown in the following example. Figure 3 shown.

[0089] Figure 3 This is a schematic diagram of the preset task interface.

[0090] Combination Figure 3 It can be seen that when the user needs to perform data processing tasks through the data management platform, he can click the "Add Task" button component provided in the task interface. The service module can display the preset file upload pop-up component to the user based on the user's click operation, so that the user can upload the original code in the file upload pop-up window and click the "Confirm Upload" button component after the upload is successful. The service module can then determine the original code entered by the user based on the user's click operation on the "Confirm Upload" button component.

[0091] In addition, the service module can display a preset text input box component to the user based on the user's click operation, so that the user can write the original code in the text input box component and upload it, and click the "Confirm Upload" button component after the upload is successful, so that the service module can determine the original code entered by the user based on the user's click operation on the "Confirm Upload" button component.

[0092] In addition, the service module can also display the execution status data of the data processing task corresponding to the target code to the user in the above-mentioned task interface, wherein the above-mentioned execution status data may include: task identification, code name, running status (such as: completed, stopped, failed, etc.), running time, etc.

[0093] Of course, the user can also query the execution status of the historically executed data processing tasks through the query button provided in the above task interface (for example, when the user needs to query the execution status data of the data processing task with the task ID 61, he can enter the status data in the preset text input box (such as Figure 3 In the dotted box), enter the task ID 61, and then click the query button to query the execution status data of the data processing task with the task ID 61), click the Delete button to delete the data processing task corresponding to the target code, use the Modify button provided in the above task interface to change the target code, use the Copy button provided in the above task interface to copy the target code, and use the Details button provided in the above task interface to query the target code.

[0094] In actual application scenarios, there may be two types of codes input by the user, the first type is the code corresponding to the stream processing task, and the second type is the code corresponding to the batch processing task. The above-mentioned stream processing task is used to process each piece of data continuously generated within a specified time period, wherein the execution module of the data management platform executes the data processing task corresponding to one piece of data each time, and then each piece of data continuously generated within the specified time period may be processed one by one (that is, within the specified time period, one piece of data is generated and the data processing task corresponding to one piece of data is executed), and the stream processing task is executed. The above-mentioned batch processing task is used to treat all the data in a static data set as a batch, and send each piece of data contained in this batch to the execution module, so that the execution module executes the data processing task corresponding to each piece of data in this batch (that is, the data processing tasks corresponding to at least part of the data in a large amount of data are batch executed by the execution module).

[0095] From the above content, it can be seen that the difference between stream processing tasks and batch processing tasks is only that the execution mode is different when executing the pending tasks corresponding to multiple data. However, the specific execution method of the pending tasks for each data is the same.

[0096] For example, if the stream processing task is "determine the order with the largest transaction amount of Company A within one day", the execution module can continuously receive the order data of Company A, and for each order data received, determine whether the transaction amount of the order data is greater than the largest transaction amount of all order data before the order data. If so, the transaction amount of the order data can be determined to be the current largest transaction amount until all order data of Company A within one day are received to determine the order with the largest transaction amount. When the batch processing task is "determine the order with the largest transaction amount of Company A within one day", the execution module will obtain an order data set consisting of all order data of Company A within one day, and process the above order data set as a batch of order data, and then compare the transaction amounts of each order data contained in the above order data set to determine the order with the largest transaction amount. Therefore, whether in the stream processing task or the batch processing task, the specific steps for comparing the transaction amounts of each order data are the same, but the execution mode is different, which leads to differences in the code of the stream processing task and the code of the batch processing task.

[0097] Therefore, in order to improve the user experience, when using the above-mentioned data management platform to perform data processing tasks, the user can upload a set of codes through the above-mentioned task interface, or write a set of codes through the text input box in the above-mentioned task interface as the original code.

[0098] Furthermore, the data management platform can generate configuration instructions through the service module based on the configuration operations performed by the user in the preset task interface, and determine the original code input by the user and the type of data processing task the user needs to perform based on the configuration instructions, and then determine the data processing strategy corresponding to the original code based on the type of data processing task the user needs to perform.

[0099] Among them, when it is determined that the type of data processing task that the user needs to perform is a stream processing task, it can be determined that the data processing strategy corresponding to the original code is the first data processing strategy, and then the original code can be converted according to the code format corresponding to the first data processing strategy to obtain the converted original code as the target code, and the target code is sent to the execution module. After the execution module receives the above-mentioned target code, it can execute the data processing task corresponding to the target code through each data processing layer in the first data link to obtain the execution result corresponding to the target code.

[0100] Furthermore, when it is determined that the type of data processing task that the user needs to perform is a batch processing task, the data processing strategy corresponding to the original code can be determined to be the second data processing strategy, and then the original code can be converted according to the code format corresponding to the second data processing strategy to obtain the converted original code as a batch processing code, and the batch processing code is sent to the execution module. When the execution model receives the above-mentioned batch processing code, it can execute the data processing task corresponding to the batch processing code through each data processing layer in the second data link to obtain the execution result corresponding to the batch processing code.

[0101] From the above content, it can be seen that when users perform data processing tasks through the data management platform, they can only upload one set of code to the service module of the data management platform, so that the service module of the data management platform can convert the original code entered by the user into code for executing stream processing tasks according to different data processing strategies, or convert it into code for executing batch processing tasks and execute it, thereby avoiding the situation where users write two sets of codes by themselves, thereby improving the user experience.

[0102] In actual application scenarios, there may be some data processing tasks corresponding to relatively complex business logic codes. In this case, the user can also click the preset file upload pop-up component in the task interface to upload a JAVA archive file (Java Archive, JAR) package containing the data processing tasks for executing the above-mentioned complex business logic. After receiving the JAR package uploaded by the user, the service module can parse the received JAR package to obtain the original code.

[0103] Furthermore, when the data processing task that the data management platform needs to execute is a stream processing task, the code corresponding to the stream processing task can be directly sent to the execution module through the service module, so that the code corresponding to the above stream processing task can be executed through the execution module. When the data processing task that the data management platform needs to execute is a batch processing task, since the batch processing task requires the data processing task of large-scale data to be handed over to the execution module for processing at one time, in order to reasonably arrange the execution order of multiple data processing tasks submitted at one time to improve the utilization rate of computing resources, a scheduling module can also be provided in the data management platform, so that the service module can send the batch processing code to the scheduling module when it determines that the data processing strategy corresponding to the original code is the second data processing strategy, and then the scheduling module can send the received batch processing code to the execution module for execution when it determines that the received batch processing code meets the preset batch processing conditions.

[0104] The above batch processing conditions may be set according to actual needs. For example, when the scheduling module determines that the priority corresponding to the received batch processing code is greater than a specified priority threshold, the above batch processing conditions may be considered to be met.

[0105] For another example, when the scheduling module determines that the idle computing resources meet the computing resource requirements required by the received batch processing code, it can be regarded as satisfying the above-mentioned batch processing conditions, etc.

[0106] Furthermore, after the execution module executes the target code and obtains the execution result corresponding to the target code, the execution result corresponding to the target code can be sent to a designated database for storage.

[0107] In actual application scenarios, in order to facilitate users to further analyze and process data, the data management platform can also send the execution results corresponding to the target code to the analysis module through the execution module, and send the execution results corresponding to the batch code to the analysis module, so that the execution results corresponding to the target code and the execution results corresponding to the batch code are displayed to the user through the analysis module. Figure 4 shown.

[0108] Figure 4 A schematic diagram of the analysis module provided in this specification.

[0109] Combination Figure 4It can be seen that the data management platform can send the execution results corresponding to the target code to the analysis module through the execution module, and send the execution results corresponding to the batch code to the analysis module, so as to receive the execution results corresponding to the target code and the execution results corresponding to the batch code through the analysis module, and reorganize the execution results corresponding to the target code and the execution results corresponding to the batch code according to the preset dimensions to obtain the reorganized execution results.

[0110] Among them, the above dimensions include: time, geographic location, data category, and user group.

[0111] For ease of understanding, this specification also provides two embodiments to facilitate detailed description of the process of the above-mentioned analysis module performing the reorganization operation.

[0112] Specifically, after viewing the total sales for each month of the year, when the user determines that the sales for a certain month (such as January) needs to be further viewed, a data reorganization request can be sent to the above-mentioned analysis module. At this time, the analysis module can reorganize the data according to the received data reorganization request to display the weekly sales, daily sales, or sales of each product per day in this month to the user.

[0113] For another example, when a user needs to determine the sales in a certain city, he or she may send a data reorganization request to the analysis module. At this time, the analysis module may reorganize the data according to the received data reorganization request to display only the sales in this city to the user.

[0114] In the above content, the analysis module may be provided with a component capable of performing complex reorganization processing operations on the execution results of batch processing tasks and the execution results of stream processing tasks, such as a hybrid online analytical processing (OLAP) component.

[0115] In addition, in order to ensure the quality of the data to be processed obtained by each data processing layer during the execution of the target code by the execution module of the data management platform, an anomaly detection module can also be provided in the data management platform so that the data management platform can detect the data to be processed through the anomaly detection module. Figure 5 shown.

[0116] Figure 5 This is a schematic diagram of the anomaly detection process provided in this specification.

[0117] Combination Figure 5It can be seen that the data management platform can run a detection task through the anomaly detection module to determine at least one data processing layer from the data processing layers run by the execution module when executing the target code as the target data processing layer, and then obtain the data to be processed corresponding to the target data processing layer, and detect the data to be processed corresponding to the target data processing layer according to the preset anomaly detection rules to obtain the detection result corresponding to the target data processing layer.

[0118] Among them, the above-mentioned anomaly detection rules are determined in advance based on the historical data output by the execution module. For example: if the historical code executed by the execution module determines the total monthly sales of a company, and after the historical code executed by the execution module, the historical data obtained through the ODS layer, that is, the sales of a company every day of the month are between 5,000 and 10,000, at this time, according to the historical data corresponding to the execution module, it can be determined that the daily sales of a company are between 5,000 and 10,000, and when the anomaly detection module determines that the data to be processed obtained by the DWD layer of the execution module (that is, the daily sales of a company) is 20,000, it can be determined that the data to be processed obtained by the DWD layer is abnormal.

[0119] Of course, the above anomaly detection rules can also be configured by the user.

[0120] For example: when the execution module executes the target code to determine a company's total sales in July, the anomaly detection rule can be set that the number of data to be processed obtained by the DWD layer should be 31. If the anomaly detection module determines that the number of data to be processed obtained by the DWD layer of the execution module is not 31, it can be determined that there are omissions in the data to be processed obtained by the DWD layer of the execution module.

[0121] Furthermore, when the data management platform detects the target data processing layer through the anomaly detection module and determines that there is no anomaly in the target data processing layer based on the detection result corresponding to the target data processing layer, a data quality report can be generated and the target code can continue to be executed through the execution module. The data quality report here is used to characterize the accuracy of the data to be processed corresponding to the target data processing layer.

[0122] When the data management platform detects the target data processing layer through the anomaly detection module and determines that there is an abnormality in the target data processing layer according to the detection result corresponding to the target data processing layer, it can determine whether it is necessary to stop executing the target code based on the detection result. If so, the execution module can be controlled to stop executing the target code, generate abnormal alarm information, and store the abnormal data to be processed corresponding to the target data processing layer, and then the above-mentioned abnormal alarm information can be displayed to the specified user in a specified manner.

[0123] If not, the execution module can be controlled to continue executing the target code. At the same time, abnormal alarm information is generated, and the abnormal data to be processed corresponding to the target data processing layer is stored, and then the above abnormal alarm information can be displayed to the specified user in a specified manner.

[0124] Among them, the method for the data management platform to determine whether it is necessary to stop executing the target code based on the detection results can be: according to the detection results, determine the exception type of the exception existing in the data to be processed corresponding to the target data processing layer; if it is determined that the exception type of the exception existing in the data to be processed corresponding to the target data processing layer is a first exception type, it is determined that the target code can continue to be executed; if it is determined that the exception type of the exception existing in the data to be processed corresponding to the target data processing layer is a second exception type, it is determined to stop executing the target code.

[0125] When the above-mentioned exception of the first exception type occurs, the execution result obtained after continuing the target code is still of reference significance, for example: the deviation value of data inconsistency is less than the preset deviation value threshold, non-critical data is missing, etc.

[0126] When the above-mentioned exception of the second exception type occurs, the execution result obtained after continuing the target code has no reference significance, for example: a large amount of data is missing, there is a conflict between data, etc.

[0127] In the above content, the designated method can be set according to actual needs, such as: instant messaging message, phone call, text message, email, etc.

[0128] From the above content, it can be seen that the data management platform can manage data according to the target code input by the user by combining components such as Mysql database, Kafka, and Hudi. Among them, the data management module can use the Mysql database to store the execution results corresponding to the target code obtained after the execution module executes the target code, thereby ensuring the accuracy and completeness of the execution results corresponding to the target code. Kafka is used as the message middleware of the execution module, which can effectively improve the data reading and writing efficiency of the execution module in the process of executing the target code, so as to improve the execution efficiency of the execution module. Hudi can store the historical data generated by the execution module in the process of executing the target code, which can facilitate the execution of backtracking operations.

[0129] For ease of understanding, this specification also provides methods executed by the execution module during data management through the above data management platform, as shown in the following example: Figure 6 shown.

[0130] Figure 6 A flowchart of a data management method provided in this specification includes the following steps:

[0131] S601: Receive target code.

[0132] S602: Run each data processing layer in the order between the data processing layers in the first data link to obtain the execution result corresponding to the target code, wherein for each data processing layer, run the data processing layer so that the data processing layer sends a data acquisition request to the first storage module to obtain the data to be processed corresponding to the data processing layer, and sends the data processed by the data processing layer as an intermediate execution result to the first storage module for storage as the data to be processed corresponding to the next data processing layer, and sends the intermediate execution result to the second storage module for storage.

[0133] S603: When it is determined that a backtracking operation needs to be performed on the target code, a historical data acquisition request is sent to the second storage module through the data processing layers in the second data link to obtain the historical data required for executing the target code from the second storage module, and the historical data is processed to obtain the processed historical data, which is the intermediate execution result generated during the execution of the target code.

[0134] From the above content, it can be seen that the data management platform can execute batch processing tasks and stream processing tasks respectively through the two data links set in the execution module, thereby avoiding the impact of offline batch processing tasks on the performance of real-time stream processing tasks.

[0135] The above are one or more data management methods of this specification. Based on the same idea, this specification also provides a corresponding data management device, such as Figure 7 shown.

[0136] Figure 7 A schematic diagram of a data management device provided in this specification includes:

[0137] Receiving module 701, used for receiving target code;

[0138] A first execution module 702 is used to run each data processing layer in the first data link in the order between the data processing layers to obtain the execution result corresponding to the target code, wherein for each data processing layer, the data processing layer is run so that the data processing layer sends a data acquisition request to the first storage module to obtain the data to be processed corresponding to the data processing layer, and sends the data processed by the data processing layer as an intermediate execution result to the first storage module for storage as the data to be processed corresponding to the next data processing layer, and sends the intermediate execution result to the second storage module for storage; and

[0139] The second execution module 703 is used to send a historical data acquisition request to the second storage module through the data processing layers in the second data link when it is determined that a backtracking operation needs to be performed on the target code, so as to obtain the historical data required for executing the target code from the second storage module, and process the historical data to obtain the processed historical data, where the historical data is the intermediate execution result generated during the execution of the target code.

[0140] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 6 A data management method is provided.

[0141] This manual also provides Figure 8 The one shown corresponds to Figure 6 A schematic diagram of the electronic device. Figure 8 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 6 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0142] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0143] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0144] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0145] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0146] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0147] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0152] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0154] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0156] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0157] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A data management platform, characterized in that: The data management platform comprises: an execution module, a first storage module, and a second storage module, wherein the execution module is provided with a first data link and a second data link, and the first data link and the second data link are mainly composed of various data processing layers; The data management platform receives the target code through the execution module, and runs each data processing layer in the order between the data processing layers in the first data link to obtain the execution result corresponding to the target code, wherein for each data processing layer, the data processing layer is run so that the data processing layer sends a data acquisition request to the first storage module to obtain the data to be processed corresponding to the data processing layer, and sends the data processed by the data processing layer as an intermediate execution result to the first storage module for storage as the data to be processed corresponding to the next data processing layer, and sends the intermediate execution result to the second storage module for storage; and When it is determined that a backtracking operation needs to be performed on the target code, a historical data acquisition request is sent to the second storage module through each data processing layer in the second data link to obtain the historical data required for executing the target code from the second storage module, and process the historical data to obtain the processed historical data, which is the intermediate execution result generated by the target code during the execution process.

2. The data management platform according to claim 1, characterized in that: The data management platform also includes: a service module; The data management platform generates configuration instructions through the service module according to the configuration operations performed by the user in the preset task interface, and determines the original code input by the user according to the configuration instructions, converts the original code into a specified format, obtains the target code, and sends the target code to the execution module.

3. The data management platform according to claim 2, characterized in that: The data management platform determines the original code input by the user and the data processing strategy corresponding to the original code according to the configuration instruction through the service module; If it is determined that the data processing strategy corresponding to the original code is the first data processing strategy, the original code is converted according to the code format corresponding to the first data processing strategy to obtain the converted original code as the target code, and the target code is sent to the execution module, and the first data processing strategy is used to perform stream processing on each piece of data that is continuously generated; The execution model is used to receive the target code, and execute the data processing task corresponding to the target code through the data processing layers in the first data link to obtain the execution result corresponding to the target code.

4. The data management platform according to claim 2, characterized in that: The data management platform determines the original code input by the user and the data processing strategy corresponding to the original code according to the configuration instruction through the service module; If it is determined that the data processing strategy corresponding to the original code is the second data processing strategy, the original code is converted according to the code format corresponding to the second data processing strategy to obtain the converted original code as a batch code, and the batch code is sent to the execution module, the second data processing strategy is used to batch process the data in the specified data set, and the amount of data contained in the specified data set remains unchanged; The execution model is used to receive the batch processing code, and execute the data processing task corresponding to the batch processing code through the data processing layers in the second data link to obtain the execution result corresponding to the batch processing code.

5. The data management platform according to claim 4, characterized in that: The data management platform also includes: a scheduling module; When the data management platform determines through the service module that the data processing strategy corresponding to the original code is the second data processing strategy, the batch processing code is sent to the scheduling module; When the scheduling module determines that the received batch processing code meets the preset batch processing condition, the received batch processing code is sent to the execution module.

6. The data management platform according to claim 4, characterized in that: The data management platform also includes: an analysis module; The data management platform sends the execution result corresponding to the target code to the analysis module through the execution module; and Sending the execution result corresponding to the batch code to the analysis module; The analysis module receives the execution result corresponding to the target code and the execution result corresponding to the batch code, and reorganizes the execution result corresponding to the target code and the execution result corresponding to the batch code according to preset dimensions to obtain the reorganized execution result, wherein the dimensions include: time, geographic location, data category, and user group.

7. The data management platform according to claim 1, characterized in that: The data management platform also includes: an anomaly detection module; The data management platform determines at least one data processing layer from the data processing layers run by the execution module when executing the target code through the anomaly detection module, as the target data processing layer, obtains the data to be processed corresponding to the target data processing layer, and detects the data to be processed corresponding to the target data processing layer according to preset anomaly detection rules to obtain the detection result corresponding to the target data processing layer, and the anomaly detection rule is determined in advance based on the historical data corresponding to the execution module.

8. The data management platform according to claim 1, characterized in that: The first storage module and the second storage module use the same logical storage layer.

9. A data management method, characterized in that: The method is applied to an execution module of a data management platform, the data management platform comprising: an execution module, a first storage module, and a second storage module, the execution module is provided with a first data link and a second data link, the first data link and the second data link are mainly composed of various data processing layers, and the method comprises: receiving object code; Running each data processing layer in the first data link in the order between the data processing layers to obtain the execution result corresponding to the target code, wherein for each data processing layer, running the data processing layer so that the data processing layer sends a data acquisition request to the first storage module to obtain the data to be processed corresponding to the data processing layer, and sends the data processed by the data processing layer as an intermediate execution result to the first storage module for storage as the data to be processed corresponding to the next data processing layer, and sends the intermediate execution result to the second storage module for storage; and When it is determined that a backtracking operation needs to be performed on the target code, a historical data acquisition request is sent to the second storage module through the data processing layers in the second data link to obtain the historical data required for executing the target code from the second storage module, and process the historical data to obtain the processed historical data, which is the intermediate execution result generated during the execution of the target code.

10. A data management device, characterized in that: The device comprises: A receiving module, used for receiving target code; a first execution module, configured to run each data processing layer in the first data link in the order between the data processing layers to obtain an execution result corresponding to the target code, wherein for each data processing layer, the data processing layer is run so that the data processing layer sends a data acquisition request to the first storage module to obtain the data to be processed corresponding to the data processing layer, and sends the data processed by the data processing layer as an intermediate execution result to the first storage module for storage as the data to be processed corresponding to the next data processing layer, and sends the intermediate execution result to the second storage module for storage; and The second execution module is used to send a historical data acquisition request to the second storage module through the data processing layers in the second data link when it is determined that a backtracking operation needs to be performed on the target code, so as to obtain the historical data required for executing the target code from the second storage module, and process the historical data to obtain the processed historical data, wherein the historical data is an intermediate execution result generated during the execution of the target code.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to claim 9 is implemented.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to claim 9 is implemented.

Images