A method and system for data management based on a logical chain

Abstract

The application discloses a data management method and system based on a logic chain, and relates to the technical field of data management.The application obtains processing nodes of a plurality of data processing tasks, forms corresponding logic chains according to execution sequences, compares the logic chains, determines same processing segments with same input data and same processing processes, executes the same processing segments to form shared intermediate data, monitors the shared intermediate data to form data monitoring results, determines usage states of the shared intermediate data in the logic chains according to the data monitoring results, and updates corresponding call relations.The application can reduce repeated execution of the same processing processes and repeated generation of the same intermediate data.

Description

Technical Field

[0001] This invention relates to the field of data management technology, specifically a data management method and system based on logical chains. Background Technology

[0002] Data processing tasks in an information system typically consist of multiple sequentially executed processing steps. Data generated in one processing step is processed by the next processing step to ultimately produce data results that meet the corresponding business requirements.

[0003] As data processing tasks in information systems continue to increase, different data processing tasks may include the same data cleaning and transformation processes. Existing data management methods typically execute the corresponding processing processes separately for each data processing task. When multiple tasks contain the same processing process, the process is executed repeatedly, generating intermediate data with identical content, thus increasing the amount of data processing. Summary of the Invention

[0004] The purpose of this invention is to provide a data management method and system based on logical chains to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a data management method based on logical chains, the data management method comprising the following steps: Step S1: Obtain the processing nodes of multiple data processing tasks, and form a logical chain corresponding to each data processing task according to the execution order of each processing node; Step S2: Compare each logic chain to identify the same processing segment in multiple logic chains that has the same input data and the same processing procedure; Step S3: Process the corresponding input data according to the same processing segment to form intermediate data, and use the intermediate data as the input data of subsequent processing nodes in each logic chain to form shared intermediate data; Step S4: Continuously monitor the shared intermediate data to generate data monitoring results; Step S5: Determine the usage status of the shared intermediate data in each logical chain according to the data monitoring results, and update the calling relationship of each logical chain to the shared intermediate data according to the usage status.

[0006] Further, step S1 specifically includes: Obtain task definition information for each data processing task from the data processing platform; the task definition information is the task flow information formed when creating or orchestrating data processing tasks, recording the processing steps included in the data processing task and the execution order of each processing step; Extract the actual data objects read by each processing stage, the data objects generated after processing, and the corresponding processing rules from the task definition information; the data objects are data units that can be independently read or written by the processing stages. Extract the data source location and object name of each data object, and combine the data source location and object name to form a data identifier corresponding to each data object. The object name is the name of the data object in the corresponding data source location. Assign node identifiers to each processing stage, and associate the node identifiers with the data identifiers of the read data objects, the data identifiers of the generated data objects, and the processing rules to form processing nodes. The processing nodes are arranged according to the execution order. When the data identifier of the data object generated by the previous processing node is the same as the data identifier of the data object read by the next processing node, the adjacent processing nodes are connected in sequence to form a logical chain corresponding to each data processing task.

[0007] Furthermore, the data processing platform is a platform capable of creating, orchestrating, and executing multiple data processing tasks. The data source location indicates the location within the data processing platform where a data object can be located and accessed. The object name is the name of the data object within its corresponding data source location; the same combination of data source location and object name corresponds to the same data object.

[0008] Further, step S2 specifically includes: Extract the processing type, the field to be processed, the processing condition, and the processing parameters from the processing rules associated with each processing node; the field to be processed is the data field called by the processing rule, the processing condition is the condition that limits the range of data to be processed, and the processing parameters are the parameters used when executing the corresponding processing type; According to the execution order of the processing nodes in each logical chain, the processing nodes that have not yet completed the comparison are selected as comparison nodes in turn. The comparison nodes are compared with the processing nodes that have not yet completed the comparison in other logical chains. The processing nodes that have the same data identifier of the read data object and the same processing type, processed field, processing condition and processing parameters are determined as the starting same nodes. Starting from the same initial node, compare the next processing node sequentially along the corresponding logical chain; when the data object read by each subsequent processing node is generated by the corresponding previous same node, and the processing type, processed field, processing condition and processing parameters of each subsequent processing node are the same, each subsequent processing node is determined as the subsequent same node, and the comparison continues until the subsequent processing node in any logical chain does not meet the above conditions. The consecutive corresponding starting nodes, subsequent identical nodes, and their connection relationships are grouped into the same processing segment. The processing nodes grouped into the processing segment are marked as having completed the comparison. The above comparison is then performed on the remaining processing nodes in each logical chain that have not yet completed the comparison, until the comparison of all processing nodes is completed. The processing segment formed by the comparison is then determined as the same processing segment.

[0009] Furthermore, the processing type indicates the data processing method performed by the processing node on the data object, including data cleaning, data filtering, data transformation, data merging, and data statistics.

[0010] For processing nodes that start from the same node, the data identifier of the data object they read is not used as a criterion for judgment; when the data read by the corresponding processing node is generated by the previous same node, and the corresponding processing type, processed field, processing condition and processing parameters are all the same, it is determined to be a subsequent same node.

[0011] Furthermore, step S3 specifically includes: Extract the data identifier of the data object read from the same starting node in each processing segment, obtain the data object corresponding to the data identifier, and use the obtained data object as the input data of the corresponding processing segment; Select one logical chain from multiple logical chains that constitute the same processing segment. According to the arrangement order of the starting node and subsequent nodes in the selected logical chain, the starting node processes the input data and uses the data object generated by the previous processing node as the data object read by the next processing node. The processing rules associated with each processing node are executed in sequence, and the data object generated by the last processing node is determined as the intermediate data. The intermediate data is stored in a shared data location that can be read by all data processing tasks, and a shared data identifier is assigned to the intermediate data to form shared intermediate data. Set the corresponding identical processing segments in each logic chain that were not selected to no longer be executed individually, determine the next processing node in each logic chain that is connected to the last processing node of the identical processing segment, replace the data identifier of the data object read by each next processing node with the shared data identifier, and make each next processing node read the shared intermediate data as input data.

[0012] Furthermore, setting the same processing segment in the unselected logical chain to no longer be executed separately means that when the corresponding data processing task is actually executed, the same processing segment is skipped, and the processing node after the same processing segment reads the shared intermediate data, while the processing nodes and their connection relationships in the original logical chain are still retained.

[0013] Furthermore, step S4 specifically includes: Extract data content that can be read individually by subsequent processing nodes from the shared intermediate data, and determine each group of individually readable data content as a data item, forming one or more data items; assign a data item identifier to each data item, extract the data content of each data item, form a corresponding content feature value according to the data content, assign an initial version identifier to the shared intermediate data, and store the initial version identifier, the data item identifier of each data item, and the content feature value accordingly; the content feature value is a unique value formed according to all the data content of the data item and changes as the data content changes; Obtain data read records formed when subsequent processing nodes in each logical chain read shared intermediate data; the data read records include node identifiers and the identifiers of the data items actually read by the corresponding processing nodes; collect the data item identifiers corresponding to the same node identifier to form a data call list for each subsequent processing node; When shared intermediate data is written again, the shared intermediate data written again is identified as the updated version, a version identifier is assigned to the updated version, and content feature values ​​are formed for each data item in the updated version; the updated version is compared with the data items with the same data item identifier in the previous version, and the data items with different content feature values ​​and the data items that exist only in one version are identified as changed data items. According to the execution order of the subsequent processing nodes in each logical chain, the data item identifier of the changed data item is compared with the data call list of each subsequent processing node in turn; when the data call list of any subsequent processing node contains the data item identifier of the changed data item, the first subsequent processing node that meets the condition is determined as the affected node of the corresponding logical chain; when all subsequent processing nodes in a logical chain do not contain the data item identifier of the changed data item, the logical chain is determined as an unaffected logical chain. The updated version identifier, changed data items, affected nodes of each logical chain, and unaffected logical chains are summarized to form the data monitoring results.

[0014] Furthermore, when the shared intermediate data can only be read as a whole by subsequent processing nodes, the shared intermediate data as a whole is determined as a single data item.

[0015] The content feature values ​​can be calculated using a data digest algorithm based on the complete data content of each data item, ensuring that identical data content results in the same content feature value, while different content feature values ​​are generated when the data content changes. The previous version refers to the most recently stored shared intermediate data version before the updated version was created. The data read record is generated by the data processing platform when subsequent processing nodes read the shared intermediate data, recording the corresponding node identifier and the identifier of the data item actually read.

[0016] In this context, the same data item uses the same data item identifier in different versions of the shared intermediate data, while newly added data items in the updated version are assigned new data item identifiers.

[0017] Furthermore, step S5 specifically includes: Based on the data monitoring results, logical chains with affected nodes are identified as being in an update and usage state, while unaffected logical chains are identified as being in a continued usage state. For logical chains that are still in use, their current call relationships to shared intermediate data remain unchanged; For a logical chain that is in the update and use state, the shared intermediate data called by the corresponding affected node is updated to the updated version, and data processing continues from the affected node according to the execution order of the processing nodes in the logical chain to complete the update of the call relationship of the corresponding logical chain.

[0018] Furthermore, the logical chains can be visualized according to the node identifiers of the processing nodes in each logical chain and the connection relationships between the processing nodes. Each processing node is displayed as a display node in the chain, and the data transmission relationship between the processing nodes is used as the connection relationship between the nodes. When multiple logical chains have the same processing segment, the same processing segment and the corresponding shared intermediate data are displayed together. When there are affected nodes in the data monitoring results, the affected nodes are marked in the corresponding logical chain, and the connection relationship between the affected nodes and the preceding and following processing nodes is displayed, thereby showing the position of the impact of the changes in shared intermediate data in the corresponding logical chain.

[0019] Furthermore, the data management system includes a logical chain construction module, a module for determining identical processing segments, a shared data formation module, a data monitoring module, and a call relationship update module; The logic chain construction module obtains the task definition information of multiple data processing tasks, forms corresponding processing nodes, and constructs the logic chain corresponding to each data processing task according to the execution order and data transmission relationship of the processing nodes. The same processing segment determination module extracts the input data and processing rules corresponding to each processing node, and continuously compares the processing nodes in different logical chains to form the same processing segment shared by multiple logical chains. The shared data forming module performs data processing corresponding to the same processing segment, shares and stores the formed intermediate data, and establishes a data call relationship between the subsequent processing nodes in each logical chain and the intermediate data to form shared intermediate data. The data monitoring module records the data calls made by each subsequent processing node to the shared intermediate data, compares the previous and current versions of the shared intermediate data, identifies the changed data items and the processing nodes in each logical chain affected by the changed data items, and forms the data monitoring results. The call relationship update module determines the usage status of the shared intermediate data for each logical chain according to the data monitoring results, updates the call relationship of the shared intermediate data for logical chains affected by the changed data items, and keeps the call relationship of the unaffected logical chains unchanged.

[0020] Furthermore, the logic chain construction module includes a task information extraction unit, a processing node formation unit, and a node connection unit; The task information extraction unit extracts the processing steps, execution order, and data objects read and generated by each processing step from the task definition information of each data processing task; the processing node forming unit associates the data objects and processing rules corresponding to each processing step with the node identifier to form a processing node; the node connection unit connects each processing node according to the execution order and the data transmission relationship between adjacent processing nodes to form a logical chain. The same processing segment determination module includes a rule information extraction unit, a start node determination unit, and a continuous node comparison unit; The rule information extraction unit extracts the processing type, processed field, processing condition, and processing parameters from the processing rules of each processing node; the starting node determination unit compares the processing nodes in different logical chains to determine the starting identical node with the same input data and processing rules; the continuous node comparison unit continues to compare subsequent processing nodes along each logical chain from the starting identical node, and forms the same processing segment by continuously satisfying the same comparison conditions and their connection relationship.

[0021] Furthermore, the shared data forming module includes an input data extraction unit, a processing segment execution unit, and a shared call setting unit; The input data extraction unit obtains data objects read from the same starting node in the same processing segment to form the input data of the same processing segment; the processing segment execution unit selects a logical chain that constitutes the same processing segment, executes the processing rules corresponding to each processing node in the same processing segment in sequence, and determines the data object generated by the last processing node as intermediate data; the shared call setting unit performs shared storage on the intermediate data and assigns a shared data identifier, and sets the corresponding subsequent processing nodes in other logical chains to read the shared intermediate data corresponding to the shared data identifier.

[0022] Furthermore, the data monitoring module includes a data item recording unit, a call recording unit, a version comparison unit, and an impact node determination unit; The data item recording unit forms one or more data items from the shared intermediate data and records the data item identifier and content feature value of each data item; the call recording unit records the data items actually read by each subsequent processing node, forming a corresponding data call list; the version comparison unit compares the updated version of the shared intermediate data with the previous version, and determines the changed data items according to the data item identifier and content feature value; the affected node determination unit compares the changed data items with the data call list of each subsequent processing node to determine the affected nodes and unaffected logical chains in each logical chain; The call relationship update module includes a usage status determination unit and a call update unit; The usage status determination unit determines the usage status of each logical chain according to the affected nodes and the unaffected logical chains; the call update unit maintains the current call relationship of the unaffected logical chains to the shared intermediate data, and updates the logical chains with affected nodes to the updated version of the shared intermediate data.

[0023] Compared with the prior art, the beneficial effects of the present invention are: 1. By comparing the logical chains corresponding to multiple data processing tasks, the same processing segments with the same input data and the same processing process are identified, and the intermediate data formed by the same processing segments are made available for multiple logical chains to call together, thereby reducing the repeated execution of the same processing process and the repeated generation of the same intermediate data.

[0024] 2. By comparing the previous and current versions of the shared intermediate data, the changed data items are identified. Combined with the data call status of each subsequent processing node, the processing node in each logical chain that is first affected by the changed data items is determined, so that the changes in the shared intermediate data can be located to the specific processing position in the corresponding logical chain.

[0025] 3. Adjust the calling relationships of each logical chain according to the impact of changes in the shared intermediate data. Maintain the original calling relationships for unaffected logical chains, and for affected logical chains, start calling the updated version from the affected node and continue processing. Reduce unnecessary duplication of processing, so that each logical chain maintains or updates the corresponding calling relationships according to the actual impact of changes in the shared intermediate data. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a data management method based on logical chains according to the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Example: Figure 1 As shown, the present invention provides a technical solution, a data management method based on a logical chain, the data management method comprising the following steps: This embodiment uses a coal mine data processing platform as an example. The safety monitoring system continuously uploads safety monitoring data to the data processing platform. The uploaded data includes the monitoring point number, acquisition time, methane concentration, and temperature. The data processing platform writes the received data into a safety monitoring raw data object and runs real-time data storage tasks, methane statistics tasks, and temperature statistics tasks on the raw safety monitoring data.

[0029] The real-time data storage task includes four sequential processing steps: raw data parsing, data format unification, invalid data cleaning, and real-time data writing; the methane statistics task includes four sequential processing steps: raw data parsing, data format unification, invalid data cleaning, and methane data statistics; and the temperature statistics task includes four sequential processing steps: raw data parsing, data format unification, invalid data cleaning, and temperature data statistics.

[0030] The data processing platform obtains the execution order of each processing step, the actual data objects read, the data objects generated after processing, and the corresponding processing rules from the task definition information of the above three data processing tasks.

[0031] Taking raw security monitoring data as an example, its data source location is the security monitoring data access location in the data processing platform, and the object name is the raw security monitoring data. The data source location and the object name are combined to form the corresponding data identifier D0.

[0032] The data processing platform assigns node identifiers A1, A2, A3, and A4 to the four processing stages in the real-time data storage task; B1, B2, B3, and B4 to the four processing stages in the methane statistics task; and C1, C2, C3, and C4 to the four processing stages in the temperature statistics task. Each node identifier is then associated with the data identifier of the data object read by the corresponding processing stage, the data identifier of the generated data object, and the processing rules.

[0033] In the real-time data storage task, A1 reads the raw safety monitoring data corresponding to D0, parses the measuring point number, acquisition time, methane concentration, and temperature, and generates a parsed data object DA1; A2 reads DA1, unifies the acquisition time format, and standardizes the data formats of methane concentration and temperature, generating a data object DA2; A3 reads DA2, removes data records lacking measuring point numbers or acquisition times, and removes duplicate data records, generating a cleaned data object DA3; A4 reads DA3 and writes or updates the corresponding real-time data according to the measuring point number and acquisition time. Since the data object generated by A1 is the data object read by A2, the data object generated by A2 is the data object read by A3, and the data object generated by A3 is the data object read by A4, a logical chain corresponding to A1-A2-A3-A4 is formed according to the above data transmission relationship.

[0034] The methane statistics task uses the same method to form a logical chain corresponding to B1-B2-B3-B4. Specifically, B1 reads D0 and generates DB1; B2 reads DB1 and generates DB2; B3 reads DB2 and generates DB3; B4 reads the measuring point number, acquisition time, and methane concentration in DB3 and performs statistics on the methane concentration at different measuring points.

[0035] The temperature statistics task forms a logical chain corresponding to C1-C2-C3-C4. Specifically, C1 reads D0 and generates DC1; C2 reads DC1 and generates DC2; C3 reads DC2 and generates DC3; C4 reads the measurement point number, acquisition time, and temperature from DC3 and performs statistical analysis on the temperature of different measurement points.

[0036] The data processing platform compares the data according to the execution order of the processing nodes in the three logical chains. A1, B1, and C1 all read the original safety monitoring data corresponding to data identifier D0. The processing type performed by all three is data transformation. The fields processed are the measurement point number, acquisition time, methane concentration, and temperature in the original safety monitoring data. The corresponding processing conditions and processing parameters are also the same. Therefore, A1, B1, and C1 are determined to be the starting nodes with the same characteristics.

[0037] Continue comparing A2, B2, and C2. Although the data identifiers of DA1, DB1, and DC1 are different, the three data objects are generated by their respective preceding identical nodes A1, B1, and C1. Furthermore, A2, B2, and C2 undergo the same data format processing, with identical processed fields, processing conditions, and processing parameters. Therefore, A2, B2, and C2 are identified as subsequent identical nodes.

[0038] Continuing to compare A3, B3, and C3, the data read by the three processing nodes were generated by A2, B2, and C2 respectively, and all of them performed the same invalid data cleaning process. The same cleaning conditions were used for missing measurement point numbers, missing acquisition time, and duplicate data records. Therefore, A3, B3, and C3 were determined to be the same nodes for the next period.

[0039] Then, A4, B4, and C4 were compared. A4 performs real-time data writing, B4 performs methane data statistics, and C4 performs temperature data statistics. The processing types and processed fields of the three are different, so the conditions for determining the same node in the future are no longer met.

[0040] The data processing platform groups A1-A2-A3, B1-B2-B3, and C1-C2-C3 and their corresponding connections into the same processing segment, and marks the processing nodes within it as having completed the comparison. It then continues to compare the remaining A4, B4, and C4. Since these three processing nodes perform different processing procedures, they do not form new identical processing segments. This completes the comparison of all processing nodes in the three logical chains mentioned above, and identifies the processing segments corresponding to A1-A2-A3, B1-B2-B3, and C1-C2-C3 as identical processing segments.

[0041] After identifying the same processing segment, the data processing platform extracts the data identifier D0 of the data objects read from the same starting nodes A1, B1, and C1, obtains the original security monitoring data corresponding to D0, and uses it as the input data for the same processing segment.

[0042] In this embodiment, the real-time data storage task is selected from three data processing tasks to execute the same processing segment. A1 parses the original security monitoring data to form DA1; A2 reads DA1 and completes the data format unification to form DA2; then A3 reads DA2 and completes invalid data cleanup, and the data object finally generated by A3 is determined as intermediate data M1.

[0043] The data processing platform stores the intermediate data M1 in a shared data location that can be read by all three data processing tasks, and assigns a shared data identifier SD1 to M1 to form shared intermediate data.

[0044] After completing the shared intermediate data setup, the next processing nodes connected to the last processing node in the same processing segment in the real-time data storage task, methane statistics task, and temperature statistics task are A4, B4, and C4, respectively. The data processing platform replaces the data identifiers of the data objects originally read by A4, B4, and C4 with the shared data identifier SD1, so that A4, B4, and C4 all read the shared intermediate data corresponding to SD1 as input data.

[0045] In this process, B1-B2-B3 and C1-C2-C3 will no longer be executed separately in subsequent tasks, but will instead skip their corresponding identical processing segments; the original logical chains of B1-B2-B3, C1-C2-C3, and their connections will still be retained. Therefore, the processes of raw data parsing, data format unification, and invalid data cleanup, which are required by all three data processing tasks, will only be executed once.

[0046] The shared intermediate data M1 includes the cleaned measurement point number, acquisition time, methane concentration, and temperature. Since the above four types of data can be read by subsequent processing nodes, this embodiment defines them as data items I1, I2, I3, and I4, respectively, where I1 corresponds to the measurement point number, I2 corresponds to the acquisition time, I3 corresponds to the methane concentration, and I4 corresponds to the temperature, and a corresponding data item identifier is assigned to each data item.

[0047] The data processing platform calculates content feature values ​​for each data item based on all its data content. This embodiment uses a data digest algorithm to calculate the content feature values. For ease of explanation, the initial content feature values ​​for I1, I2, I3, and I4 are denoted as H1, H2, H3, and H4, respectively. Content feature values ​​are formed based on all the data content of the corresponding data item. The same content feature value is formed when the data content of a data item remains unchanged; different content feature values ​​are formed when the data content changes.

[0048] The data processing platform assigns an initial version identifier V1 to the shared intermediate data generated this time, and stores V1, the data item identifiers of each data item, and the corresponding content feature values ​​in association.

[0049] As the three data processing tasks continue to run, the data processing platform synchronously records the actual reading status of the shared intermediate data by each subsequent processing node.

[0050] When A4 performs real-time data writing, it reads the measurement point number, acquisition time, methane concentration, and temperature. Therefore, the data reading record formed by A4 includes the node identifier A4 and the data item identifiers I1, I2, I3, and I4.

[0051] When performing methane data statistics, B4 reads the measurement point number, collection time, and methane concentration. Therefore, the data reading record generated by B4 includes the node identifier B4 and the data item identifiers I1, I2, and I3.

[0052] When performing temperature data statistics, C4 reads the measurement point number, collection time, and temperature. Therefore, the data reading record generated by C4 includes the node identifier C4 and the data item identifiers I1, I2, and I4.

[0053] The data processing platform aggregates the data item identifiers in the above data reading records according to the node identifier, thereby forming the data call lists corresponding to A4, B4 and C4 respectively.

[0054] Subsequently, the safety monitoring system corrected a batch of uploaded safety monitoring data. In one of the safety monitoring records, the methane concentration was corrected from 0.42% to 0.47%. The corresponding measuring point number and collection time remained unchanged, and the temperature data also remained unchanged.

[0055] The corrected raw security monitoring data undergoes the same processing steps as A1, A2, and A3 again, forming new intermediate data which is then written back to the shared data location. The data processing platform identifies this newly written shared intermediate data as the updated version and assigns it a version identifier V2. At this point, V1 is the most recently stored version of the shared intermediate data before V2 was formed, i.e., the previous version corresponding to V2.

[0056] The data processing platform generates content feature values ​​for I1, I2, I3, and I4 in V2 in the same manner as V1. Since the measurement point number remains unchanged, the content feature value for I1 is still H1; since the acquisition time remains unchanged, the content feature value for I2 is still H2; because the methane concentration was corrected from 0.42% to 0.47%, the data content of I3 has changed, and its content feature value has changed from H3 to H3′; since the temperature data remains unchanged, the content feature value for I4 is still H4.

[0057] After comparing the data items with the same data item identifier in V2 and V1 one by one, the content feature values ​​of I1, I2 and I4 are all the same, while the content feature value of I3 is different. Therefore, I3 is identified as the changing data item.

[0058] When new data items are added during subsequent updates, or when data items that existed in the previous version no longer exist in the updated version, these data items are also identified as changed data items because they only existed in one version.

[0059] After determining the changed data item I3, the data processing platform compares the data item identifier of I3 with the data call list of the corresponding processing node according to the execution order of the subsequent processing nodes in each logical chain.

[0060] In the logical chain corresponding to the real-time data storage task, the data call list of A4 contains I3. Therefore, A4 is the first subsequent processing node in the logical chain that meets the conditions, and A4 is identified as the affected node corresponding to the real-time data storage task.

[0061] In the logical chain corresponding to the methane statistics task, the data call list of B4 contains I3, so B4 is identified as the affected node corresponding to the methane statistics task.

[0062] In the logical chain corresponding to the temperature statistics task, the data call list of C4 includes I1, I2 and I4, but does not include the changed data item I3. Furthermore, there are no other subsequent processing nodes that read I3 in this logical chain. Therefore, the logical chain corresponding to the temperature statistics task is determined to be an unaffected logical chain.

[0063] The data processing platform will summarize the updated version identifier V2, the changed data item I3, the affected node A4 of the real-time data storage task, the affected node B4 of the methane statistics task, and the corresponding logical chain of the unaffected temperature statistics task to form the data monitoring results corresponding to the shared intermediate data.

[0064] Based on the above data monitoring results, the data processing platform determined that the real-time data storage task with affected node A4 and the methane statistics task with affected node B4 should be updated and put into use, while the temperature statistics task should be put into use.

[0065] For real-time data storage tasks that are in the update and usage state, the shared intermediate data called by A4 is updated from the previous version V1 to the updated version V2, and real-time data writing continues from A4, so that the corrected methane concentration data is written to the corresponding real-time data results.

[0066] For methane statistics tasks that are in the update and use state, the shared intermediate data called by B4 will be updated from V1 to V2, and methane data statistics will be re-performed starting from B4, so that the 0.47% corrected methane concentration will participate in the corresponding statistical processing.

[0067] For temperature statistics tasks that are still in use, since the I1, I2 and I4 actually read by the subsequent processing node C4 have not changed, the call relationship between C4 and the shared intermediate data of the previous version V1 remains unchanged, and temperature data statistics are not re-executed.

[0068] Therefore, when the content of the shared intermediate data changes, it is not necessary to re-execute the same processing segment shared by the three tasks, nor is it necessary to re-execute the temperature statistics task that has not read the changed data items. Instead, only the corresponding call relationship of the logical chain that is actually affected by the changed data items is updated.

[0069] Furthermore, the data processing platform can also visualize the three logical chains mentioned above according to the node identifiers and connection relationships of each processing node. A1 to A4, B1 to B4, and C1 to C4 serve as display nodes in the corresponding logical chains, and the data transmission relationships between processing nodes serve as the connection relationships between adjacent display nodes.

[0070] For the same processing segment formed by A1-A2-A3, B1-B2-B3, and C1-C2-C3, it is displayed in the visualization interface in association with the shared intermediate data SD1. After the updated version V2 is formed and data monitoring is completed, the affected node A4 is marked in the logical chain corresponding to the real-time data storage task, and the affected node B4 is marked in the logical chain corresponding to the methane statistics task. At the same time, the connection relationship between the affected node and the corresponding previous processing node is displayed. The affected node of the temperature statistics task has not been determined, so the original logical chain display state is maintained, so as to show the actual impact position of the change in the shared intermediate data in different data processing tasks.

[0071] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data management method based on logical chains, characterized in that: The data management method includes the following steps: Step S1: Obtain the processing nodes of multiple data processing tasks, and form a logical chain corresponding to each data processing task according to the execution order of each processing node; Step S2: Compare each logic chain to identify the same processing segment in multiple logic chains that has the same input data and the same processing procedure; Step S3: Process the corresponding input data according to the same processing segment to form intermediate data, and use the intermediate data as the input data of subsequent processing nodes in each logic chain to form shared intermediate data; Step S4: Continuously monitor the shared intermediate data to generate data monitoring results; Step S5: Determine the usage status of the shared intermediate data in each logical chain according to the data monitoring results, and update the calling relationship of each logical chain to the shared intermediate data according to the usage status.

2. The data management method based on logical chains according to claim 1, characterized in that: Step S1 specifically includes: Obtain task definition information for each data processing task from the data processing platform; the task definition information is the task flow information formed when creating or orchestrating data processing tasks, recording the processing steps included in the data processing task and the execution order of each processing step; Extract the actual data objects read by each processing stage, the data objects generated after processing, and the corresponding processing rules from the task definition information; the data objects are data units that can be independently read or written by the processing stages. Extract the data source location and object name of each data object, and combine the data source location and object name to form a data identifier corresponding to each data object. The object name is the name of the data object in the corresponding data source location. Assign node identifiers to each processing stage, and associate the node identifiers with the data identifiers of the read data objects, the data identifiers of the generated data objects, and the processing rules to form processing nodes. The processing nodes are arranged according to the execution order. When the data identifier of the data object generated by the previous processing node is the same as the data identifier of the data object read by the next processing node, the adjacent processing nodes are connected in sequence to form a logical chain corresponding to each data processing task.

3. The data management method based on logical chains according to claim 2, characterized in that: Step S2 specifically includes: Extract the processing type, the field to be processed, the processing condition, and the processing parameters from the processing rules associated with each processing node; the field to be processed is the data field called by the processing rule, the processing condition is the condition that limits the range of data to be processed, and the processing parameters are the parameters used when executing the corresponding processing type; According to the execution order of the processing nodes in each logical chain, the processing nodes that have not yet completed the comparison are selected as comparison nodes in turn. The comparison nodes are compared with the processing nodes that have not yet completed the comparison in other logical chains. The processing nodes that have the same data identifier of the read data object and the same processing type, processed field, processing condition and processing parameters are determined as the starting same nodes. Starting from the same initial node, compare the next processing node sequentially along the corresponding logical chain; when the data object read by each subsequent processing node is generated by the corresponding previous same node, and the processing type, processed field, processing condition and processing parameters of each subsequent processing node are the same, each subsequent processing node is determined as the subsequent same node, and the comparison continues until the subsequent processing node in any logical chain does not meet the above conditions. The consecutive corresponding starting nodes, subsequent identical nodes, and their connection relationships are grouped into the same processing segment. The processing nodes grouped into the processing segment are marked as having completed the comparison. The above comparison is then performed on the remaining processing nodes in each logical chain that have not yet completed the comparison, until the comparison of all processing nodes is completed. The processing segment formed by the comparison is then determined as the same processing segment.

4. The data management method based on logical chains according to claim 3, characterized in that: Step S3 specifically includes: Extract the data identifier of the data object read from the same starting node in each processing segment, obtain the data object corresponding to the data identifier, and use the obtained data object as the input data of the corresponding processing segment; Select one logical chain from multiple logical chains that constitute the same processing segment. According to the arrangement order of the starting node and subsequent nodes in the selected logical chain, the starting node processes the input data and uses the data object generated by the previous processing node as the data object read by the next processing node. The processing rules associated with each processing node are executed in sequence, and the data object generated by the last processing node is determined as the intermediate data. The intermediate data is stored in a shared data location that can be read by all data processing tasks, and a shared data identifier is assigned to the intermediate data to form shared intermediate data. Set the corresponding identical processing segments in each logic chain that were not selected to no longer be executed individually, determine the next processing node in each logic chain that is connected to the last processing node of the identical processing segment, replace the data identifier of the data object read by each next processing node with the shared data identifier, and make each next processing node read the shared intermediate data as input data.

5. The data management method based on logical chains according to claim 1, characterized in that: Step S4 specifically includes: Extract data content that can be read individually by subsequent processing nodes from the shared intermediate data, and determine each group of individually readable data content as a data item, forming one or more data items; assign a data item identifier to each data item, extract the data content of each data item, form a corresponding content feature value according to the data content, assign an initial version identifier to the shared intermediate data, and store the initial version identifier, the data item identifier of each data item, and the content feature value accordingly; the content feature value is a unique value formed according to all the data content of the data item and changes as the data content changes; Obtain data read records formed when subsequent processing nodes in each logical chain read shared intermediate data; the data read records include node identifiers and the identifiers of the data items actually read by the corresponding processing nodes; collect the data item identifiers corresponding to the same node identifier to form a data call list for each subsequent processing node; When shared intermediate data is written again, the shared intermediate data written again is identified as the updated version, a version identifier is assigned to the updated version, and content feature values ​​are formed for each data item in the updated version; the updated version is compared with the data items with the same data item identifier in the previous version, and the data items with different content feature values ​​and the data items that exist only in one version are identified as changed data items. According to the execution order of the subsequent processing nodes in each logical chain, the data item identifier of the changed data item is compared with the data call list of each subsequent processing node in turn; when the data call list of any subsequent processing node contains the data item identifier of the changed data item, the first subsequent processing node that meets the condition is determined as the affected node of the corresponding logical chain; when all subsequent processing nodes in a logical chain do not contain the data item identifier of the changed data item, the logical chain is determined as an unaffected logical chain. The updated version identifier, changed data items, affected nodes of each logical chain, and unaffected logical chains are summarized to form the data monitoring results.

6. The data management method based on logical chains according to claim 5, characterized in that: Step S5 specifically includes: Based on the data monitoring results, logical chains with affected nodes are identified as being in an update and usage state, while unaffected logical chains are identified as being in a continued usage state. For logical chains that are still in use, their current call relationships to shared intermediate data remain unchanged; For a logical chain that is in the update and use state, the shared intermediate data called by the corresponding affected node is updated to the updated version, and data processing continues from the affected node according to the execution order of the processing nodes in the logical chain to complete the update of the call relationship of the corresponding logical chain.

7. A data management system based on logical chains, applied to the data management method based on logical chains as described in any one of claims 1-6, characterized in that: The data management system includes a logical chain construction module, a same processing segment determination module, a shared data formation module, a data monitoring module, and a call relationship update module; The logic chain construction module obtains the task definition information of multiple data processing tasks, forms corresponding processing nodes, and constructs the logic chain corresponding to each data processing task according to the execution order and data transmission relationship of the processing nodes. The same processing segment determination module extracts the input data and processing rules corresponding to each processing node, and continuously compares the processing nodes in different logical chains to form the same processing segment shared by multiple logical chains. The shared data forming module performs data processing corresponding to the same processing segment, shares and stores the formed intermediate data, and establishes a data call relationship between the subsequent processing nodes in each logical chain and the intermediate data to form shared intermediate data. The data monitoring module records the data calls made by each subsequent processing node to the shared intermediate data, compares the previous and current versions of the shared intermediate data, identifies the changed data items and the processing nodes in each logical chain affected by the changed data items, and forms the data monitoring results. The call relationship update module determines the usage status of the shared intermediate data for each logical chain according to the data monitoring results, updates the call relationship of the shared intermediate data for logical chains affected by the changed data items, and keeps the call relationship of the unaffected logical chains unchanged.

8. A data management system based on logical chains according to claim 7, characterized in that: The logical chain construction module includes a task information extraction unit, a processing node formation unit, and a node connection unit; The task information extraction unit extracts the processing steps, execution order, and data objects read and generated by each processing step from the task definition information of each data processing task; the processing node forming unit associates the data objects and processing rules corresponding to each processing step with the node identifier to form a processing node; the node connection unit connects each processing node according to the execution order and the data transmission relationship between adjacent processing nodes to form a logical chain. The same processing segment determination module includes a rule information extraction unit, a start node determination unit, and a continuous node comparison unit; The rule information extraction unit extracts the processing type, the field being processed, the processing conditions, and the processing parameters from the processing rules of each processing node; the starting node determination unit compares the processing nodes in different logical chains to determine the starting node with the same input data and processing rules. The continuous node comparison unit continues to compare subsequent processing nodes along each logical chain from the starting same node, and forms the same processing segment by continuously satisfying the same comparison conditions and their connection relationship.

9. A data management system based on logical chains according to claim 7, characterized in that: The shared data formation module includes an input data extraction unit, a processing segment execution unit, and a shared call setting unit; The input data extraction unit obtains data objects read from the same starting node in the same processing segment to form the input data of the same processing segment; the processing segment execution unit selects a logical chain that constitutes the same processing segment, executes the processing rules corresponding to each processing node in the same processing segment in sequence, and determines the data object generated by the last processing node as intermediate data; the shared call setting unit performs shared storage on the intermediate data and assigns a shared data identifier, and sets the corresponding subsequent processing nodes in other logical chains to read the shared intermediate data corresponding to the shared data identifier.

10. A data management system based on logical chains according to claim 7, characterized in that: The data monitoring module includes a data item recording unit, a call recording unit, a version comparison unit, and an impact node determination unit. The data item recording unit forms one or more data items from the shared intermediate data and records the data item identifier and content feature value of each data item; the call recording unit records the data items actually read by each subsequent processing node, forming a corresponding data call list; the version comparison unit compares the updated version of the shared intermediate data with the previous version, and determines the changed data items according to the data item identifier and content feature value; the affected node determination unit compares the changed data items with the data call list of each subsequent processing node to determine the affected nodes and unaffected logical chains in each logical chain; The call relationship update module includes a usage status determination unit and a call update unit; The usage status determination unit determines the usage status of each logical chain according to the affected nodes and the unaffected logical chains; the call update unit maintains the current call relationship of the unaffected logical chains to the shared intermediate data, and updates the logical chains with affected nodes to the updated version of the shared intermediate data.

Images