A Big Data-Driven End-to-End Monitoring Method and System Based on Intelligent Analysis

Abstract

This invention discloses a big data end-to-end supervision method and system based on intelligent analysis, comprising the following steps: acquiring the end-to-end processing events and regulatory attribute information set of real data objects in the big data platform; constructing a regulatory shadow object corresponding to the real data object; structurally encoding the end-to-end processing events to generate a sequence of real data behavior trajectories; driving the regulatory shadow object to perform regulatory state migration and generating a regulatory shadow state migration chain; constructing an expected compliance migration chain corresponding to the real data object; performing shadow chain offset alignment processing on the regulatory shadow state migration chain and the expected compliance migration chain; determining the offset information, generating the big data end-to-end supervision result, and outputting the corresponding regulatory handling strategy. This invention utilizes the regulatory shadow state migration analysis method to achieve dynamic end-to-end supervision of big data, possessing advantages such as high regulatory accuracy, strong anomaly identification capability, and high handling efficiency.

Description

Technical Field

[0001] This invention relates to the field of data compliance supervision, and in particular to a method and system for the whole-process supervision of big data based on intelligent analysis. Background Technology

[0002] With the widespread application of big data platforms in government, finance, healthcare, and industry, the flow of real data objects during the processes of collection, cleaning, fusion, storage, retrieval, processing, sharing, export, archiving, and destruction is constantly increasing, and the relationships between data processing nodes, processing entities, and data flow are becoming increasingly complex. Existing big data supervision technologies typically monitor data flow behavior through log auditing, permission verification, or rule matching to achieve data access control, abnormal behavior identification, and compliance auditing.

[0003] Most existing technologies rely on static rules or single-node behavior for regulatory analysis, lacking continuous correlation modeling of the entire process of real data object state migration. This makes it difficult to accurately identify hidden deviations in data flow across nodes. Furthermore, existing technologies lack the ability to dynamically align the expected compliance state chain with the actual processing state chain, resulting in the difficulty in timely detecting abnormal behaviors such as data usage deviations, flow deviations, anonymization deviations, and lifecycle deviations. This, in turn, affects the accuracy and efficiency of the entire process of big data platform supervision.

[0004] Therefore, how to provide a big data end-to-end supervision method and system based on intelligent analysis is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a big data full-process supervision method and system based on intelligent analysis. This invention utilizes the supervision shadow state migration analysis method to achieve dynamic supervision of the entire big data process, and has the advantages of high supervision accuracy, strong anomaly identification capability and high handling efficiency.

[0006] A big data end-to-end monitoring method based on intelligent analysis according to an embodiment of the present invention includes the following steps: Acquire the full-process processing events of real data objects in the big data platform and the corresponding regulatory attribute information set; Construct regulatory shadow objects that correspond to real data objects based on a set of regulatory attribute information; The entire process of events is structured and encoded according to the event occurrence time, processing node identifier, processing subject identifier, processing action type, input data identifier, output data identifier, and target flow direction identifier to generate a sequence of real data behavior trajectories. Based on the processing action types in the real data behavior trajectory sequence and the regulatory constraint information in the regulatory shadow object, the regulatory shadow object is driven to perform regulatory state transition and generate a regulatory shadow state transition chain. Based on business process templates and regulatory constraint information, the scope of process nodes and the boundaries of compliance constraints are determined. The historical compliance and regulatory status sequence is feature-encoded, and a compliance benchmark sequence is generated through the Informer model to construct the expected compliance migration chain. Perform shadow chain offset alignment processing on the regulatory shadow state migration chain and the expected compliance migration chain to generate a regulatory offset feature set; Based on the set of regulatory offset features, offset information is determined, big data full-process regulatory results are generated, and corresponding regulatory handling strategies are output.

[0007] Optionally, the full-process processing events are processing events generated during the collection, cleaning, fusion, storage, retrieval, processing, sharing, export, archiving, and destruction of real data objects.

[0008] Optionally, the construction of the regulatory shadow object specifically includes: The data attribute information, business attribute information, permission attribute information, compliance attribute information, and lifecycle attribute information in the regulatory attribute information set are processed to standardize fields, unify identifiers, and normalize timestamps to generate a standardized set of regulatory attributes. Based on the data attribute information and business attribute information in the standardized regulatory attribute set, generate real data object identifiers, data source identifiers and regulatory identifiers, and establish a one-to-one correspondence between regulatory identifiers and real data object identifiers; A regulatory shadow object structure template is constructed based on a standardized set of regulatory attributes. The regulatory shadow object structure template does not set a field for storing real data content, but is only used to carry the regulatory identifier, real data object identifier, data source identifier, current regulatory status identifier and regulatory constraint information corresponding to the real data object. Based on the data attribute information and compliance attribute information in the standardized regulatory attribute set, determine the sensitivity level identifier and de-identification requirement identifier corresponding to the real data object; based on the business attribute information, permission attribute information and compliance attribute information, determine the authorized use identifier, access permission boundary identifier, permitted flow identifier and processing constraint identifier corresponding to the real data object; based on the life cycle attribute information, determine the retention period identifier corresponding to the real data object. The sensitivity level identifier, authorized use identifier, access permission boundary identifier, permitted flow identifier, processing constraint identifier, desensitization requirement identifier, and retention period identifier are merged into the regulatory constraint information in the regulatory shadow object; Write the regulatory identifier, real data object identifier, data source identifier, current regulatory status identifier, and regulatory constraint information into the regulatory shadow object structure template to generate a regulatory shadow object.

[0009] Optionally, the generation of the real data behavior trajectory sequence specifically includes: The event fields of the entire process are parsed to extract the event occurrence time, processing node identifier, processing subject identifier, processing action type, input data identifier, output data identifier, and target flow identifier; The event occurrence time is uniformly processed at the time granularity to generate an event time code. Then, category mapping processing is performed on the processing node identifier, processing subject identifier, processing action type, input data identifier, output data identifier, and target flow direction identifier to generate processing node code, processing subject code, processing action code, input data code, output data code, and target flow direction code. A single-event structured coding unit is constructed based on event time coding, processing node coding, processing subject coding, processing action coding, input data coding, output data coding, and target flow coding; Multiple single-event structured coding units corresponding to the same real data object are sequentially ordered according to the time of event occurrence. Data flow connection relationships between sequentially related single-event structured coding units are established based on input data identifiers, output data identifiers, and target flow direction identifiers. The sequence of real data behavior trajectories is composed of multiple sequentially ordered single-event structured coding units and the data flow connection relationships between sequentially related single-event structured coding units.

[0010] Optionally, the generation of the regulatory shadow state transition chain specifically includes: Based on the processing action type in the real data behavior trajectory sequence, determine the regulatory status type to be migrated corresponding to the regulatory shadow object, and perform compliance constraint verification on the processing action type, processing subject identifier, processing node identifier and target flow identifier based on the regulatory constraint information in the regulatory shadow object to obtain the regulatory status migration node; Multiple regulatory state transition nodes are sequentially sorted according to the event occurrence time in the real data behavior trajectory sequence, and the state succession relationship between the regulatory state transition nodes is established based on the input data identifier, output data identifier, and target flow identifier. The multiple regulatory state transition nodes after time-series sorting and the state succession relationships between the preceding and following regulatory state transition nodes are encapsulated in a chain to generate a regulatory shadow state transition chain corresponding to the real data behavior trajectory sequence.

[0011] Optionally, the construction of the expected compliance migration chain specifically includes: Analyze the process node identifiers, node execution order, node input requirements, node output requirements, and node transition conditions in the business process template to determine the scope of process nodes in the expected compliance migration chain; Based on the regulatory constraint information in the regulatory rule base and regulatory shadow objects, constraints are imposed on the processing subject, processing action, data flow, processing method, desensitization status, life cycle status and output result status in the scope of process nodes to determine the compliance constraint boundaries of the expected compliance migration chain. Extract historical compliance regulatory status sequences from historical compliance processing samples, and encode the historical processing nodes, historical processing subjects, historical processing actions, historical data flow, historical processing methods, historical desensitization status, historical life cycle status, and historical output result status in the historical compliance regulatory status sequences to generate historical compliance regulatory status feature sequences. The historical compliance and regulatory status feature sequence is input into the Informer model. Temporal feature extraction and regulatory status association modeling are performed through time location encoding layer, encoder layer, probabilistic sparse attention layer and decoder layer to predict the compliance benchmark sequence corresponding to the real data object. The compliance benchmark sequence is decoded to obtain the benchmark processing node, benchmark processing subject, benchmark processing action, benchmark data flow, benchmark processing method, benchmark desensitization status, benchmark life cycle status, and benchmark output result status. Within the scope of process nodes and compliance constraint boundaries, the decoded benchmark statuses are chained together according to the status sequence to construct the expected compliance migration chain corresponding to the real data object.

[0012] Optionally, the generation of the regulatory offset feature set specifically includes: Using real data object identifiers, regulatory identifiers, and the order of status occurrence as a unified alignment benchmark, an inter-chain node index relationship is established for the regulatory shadow status migration chain and the expected compliance migration chain. Based on the inter-chain node index relationship, the processing node identifier in the regulatory status migration node is aligned with the expected processing node in the expected compliance migration chain at the node level to obtain the node-level alignment result. Under the node-level alignment results, the processing action type, target flow direction identifier, and actual use identifier are matched with the expected processing action, expected data flow direction, and permitted use constraints, respectively. Based on the access permission boundary identifier, de-identification requirement identifier, and retention period identifier, the processing subject identifier, de-identification status, and lifecycle status are matched with the expected processing subject, expected de-identification status, and expected lifecycle status, respectively. Based on the output data identifier, target flow identifier, and post-migration regulatory status identifier, the output result status in the regulatory shadow status migration chain is aligned with the expected output result status at the output result level. Based on the alignment results at the node level, action level, flow level, purpose level, permission level, desensitization level, lifecycle level, and output result level, determine the offset fields, offset order, and offset range of the regulatory shadow state migration chain relative to the expected compliance migration chain. The offset fields, offset order, and offset range are respectively merged into the permission offset feature, purpose offset feature, flow offset feature, processing offset feature, desensitization offset feature, lifecycle offset feature, and output result offset feature to generate a regulatory offset feature set.

[0013] Optionally, the generation of the big data full-process supervision results and supervision and handling strategies specifically includes: Based on the offset field, offset order and offset range in the regulatory offset feature set, the corresponding regulatory state migration node is located, the processing node identifier, processing subject identifier, processing action type and target flow identifier are extracted, and regulatory state migration nodes with the same offset field and continuous offset range are merged to generate offset nodes. Based on the categories of permission offset features, purpose offset features, flow offset features, processing offset features, desensitization offset features, lifecycle offset features, and output result offset features, determine the offset type corresponding to the offset node; The offset magnitude corresponding to the offset node is determined based on the number of offset fields within the same offset node, the constraint strength corresponding to the regulatory constraint information, and the offset range. The position of each offset node in the regulatory shadow state migration chain is determined according to the offset order. The first offset node is determined as the offset starting position, and the offset influence range is determined according to the offset range. The offset node, offset type, offset magnitude, offset starting position, and offset impact range are associated and encapsulated with the real data object identifier, regulatory identifier, and target flow identifier to generate big data full-process regulatory results. Based on the offset type and offset magnitude in the big data full-process supervision results, corresponding regulatory handling conditions are matched, and corresponding regulatory handling strategies are generated based on the matched regulatory handling conditions. The regulatory handling strategies include permission convergence strategy, sharing restriction strategy, de-identification enhancement strategy, anomaly export blocking strategy, data rollback strategy, lifecycle compression strategy, and audit enhancement strategy.

[0014] According to an embodiment of the present invention, a big data end-to-end monitoring system based on intelligent analysis includes: The event acquisition module is used to acquire a set of information on events and regulatory attributes throughout the entire process. The shadow construction module is used to construct regulatory shadow objects that correspond to real data objects; The trajectory generation module is used to generate real data behavior trajectory sequences for the entire process of structured coding events. The state transition module is used to drive the regulatory shadow object to perform regulatory state transitions and generate a regulatory shadow state transition chain. The compliance building module is used to build the expected compliance migration chain corresponding to the real data objects; The offset analysis module is used to perform shadow chain offset alignment processing and generate a set of regulatory offset features. The results output module is used to determine offset information, generate big data full-process supervision results, and output supervision and handling strategies.

[0015] The beneficial effects of this invention are: This invention achieves decoupling management of real data content and the regulatory process by constructing regulatory shadow objects corresponding to real data objects and utilizing these shadow objects to carry regulatory identifiers, regulatory status identifiers, and regulatory constraint information. Without the real data objects directly participating in regulatory analysis, continuous monitoring of the entire data processing behavior throughout the entire process of data collection, cleaning, fusion, storage, retrieval, processing, sharing, export, archiving, and destruction is possible, reducing the data leakage risk associated with direct reading of real data content in traditional regulatory methods. Simultaneously, by uniformly mapping regulatory constraint information to the regulatory shadow objects, the regulatory boundaries corresponding to different processing nodes, processing entities, and target flows are uniformly constrained, improving the consistency and standardization of the entire process of big data platform supervision.

[0016] This invention constructs a sequence of real data behavior trajectories by structurally encoding all processing events throughout the entire process, and further generates a regulatory shadow state migration chain, achieving chain-like correlation analysis of the entire process of real data object state changes. Compared to traditional regulatory methods based on single log audits or static rule verification, this invention can continuously track the state continuity relationship during data flow by combining event occurrence time, processing action type, input data identifier, output data identifier, and target flow direction identifier, improving the ability to identify abnormal flow behavior across nodes. Simultaneously, by constructing an expected compliance migration chain and performing shadow chain offset alignment processing between the regulatory shadow state migration chain and the expected compliance migration chain, the system can identify various abnormal behaviors such as permission offsets, usage offsets, flow direction offsets, processing offsets, desensitization offsets, lifecycle offsets, and output result offsets, improving the accuracy of supervision over hidden violations.

[0017] This invention generates end-to-end big data regulatory results based on a set of regulatory offset features, and generates corresponding regulatory handling strategies by combining offset type and magnitude, achieving coordinated processing from anomaly identification to regulatory action. For different scenarios such as permission anomalies, sharing anomalies, anonymization anomalies, anomaly export anomalies, and lifecycle anomalies, the system can generate permission convergence strategies, sharing restriction strategies, anonymization enhancement strategies, anomaly export blocking strategies, data rollback strategies, lifecycle compression strategies, and audit enhancement strategies, thereby improving the automated handling capabilities of the big data platform for abnormal behavior. Simultaneously, by analyzing the offset order, offset range, and offset impact scope, it can accurately locate the propagation chain of abnormal behavior, improve regulatory response efficiency, and enhance the compliance, security, and traceability of the entire big data platform operation. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a big data end-to-end monitoring method based on intelligent analysis proposed in this invention; Figure 2 This is a flowchart illustrating the expected compliance migration chain construction of a big data end-to-end regulatory method based on intelligent analysis proposed in this invention. Figure 3 This is a flowchart illustrating the generation of regulatory offset features for a big data-based end-to-end regulatory method proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figures 1-3 A big data-driven end-to-end monitoring method based on intelligent analysis includes the following steps: Acquire the full-process processing events of real data objects in the big data platform and the corresponding regulatory attribute information set; Construct regulatory shadow objects that correspond to real data objects based on a set of regulatory attribute information; The entire process of events is structured and encoded according to the event occurrence time, processing node identifier, processing subject identifier, processing action type, input data identifier, output data identifier, and target flow direction identifier to generate a sequence of real data behavior trajectories. Based on the processing action types in the real data behavior trajectory sequence and the regulatory constraint information in the regulatory shadow object, the regulatory shadow object is driven to perform regulatory state transition and generate a regulatory shadow state transition chain. Based on business process templates and regulatory constraint information, the scope of process nodes and the boundaries of compliance constraints are determined. The historical compliance and regulatory status sequence is feature-encoded, and a compliance benchmark sequence is generated through the Informer model to construct the expected compliance migration chain. Perform shadow chain offset alignment processing on the regulatory shadow state migration chain and the expected compliance migration chain to generate a regulatory offset feature set; Based on the set of regulatory offset features, offset information is determined, big data full-process regulatory results are generated, and corresponding regulatory handling strategies are output.

[0021] In this embodiment, the full-process processing events are the processing events generated by real data objects during the processes of collection, cleaning, fusion, storage, retrieval, processing, sharing, export, archiving, and destruction.

[0022] In this embodiment, the construction of the regulatory shadow object specifically includes: The data attribute information, business attribute information, permission attribute information, compliance attribute information, and lifecycle attribute information in the regulatory attribute information set are processed to standardize fields, unify identifiers, and normalize timestamps to generate a standardized set of regulatory attributes. Based on the data attribute information and business attribute information in the standardized regulatory attribute set, a real data object identifier, a data source identifier, and a regulatory identifier are generated, and a one-to-one correspondence between the regulatory identifier and the real data object identifier is established. Specifically, the following steps are taken: data object name, data object type, data table identifier, field identifier, file identifier, interface identifier, and data version identifier are extracted from the data attribute information; business system identifier, business process identifier, business scenario identifier, and business responsible entity identifier are extracted from the business attribute information; the data object name, data object type, data table identifier, field identifier, file identifier, interface identifier, and data version identifier are combined and encoded to generate the real data object identifier; the business system identifier, business process identifier, business scenario identifier, and business responsible entity identifier are source-encoded to generate the data source identifier; the real data object identifier, data source identifier, and regulatory generation time identifier are associated and encoded to generate the regulatory identifier; and the regulatory identifier and the real data object identifier are written into the regulatory shadow object mapping table to establish a one-to-one correspondence between the regulatory identifier and the real data object identifier. A regulatory shadow object structure template is constructed based on a standardized set of regulatory attributes. The regulatory shadow object structure template does not set fields for storing real data content, but is only used to carry the regulatory identifier, real data object identifier, data source identifier, current regulatory status identifier, and regulatory constraint information corresponding to the real data object. Among them, the regulatory identifier is used to distinguish different regulatory shadow objects, the real data object identifier is used to point to the corresponding real data object, the data source identifier is used to characterize the source system or source node of the real data object, and the current regulatory status identifier is used to record the regulatory status of the regulatory shadow object in the entire process. Based on the data attribute information and compliance attribute information in the standardized regulatory attribute set, determine the sensitivity level identifier and de-identification requirement identifier corresponding to the real data object; based on the business attribute information, permission attribute information and compliance attribute information, determine the authorized use identifier, access permission boundary identifier, permitted flow identifier and processing constraint identifier corresponding to the real data object; based on the life cycle attribute information, determine the retention period identifier corresponding to the real data object. Specifically, this involves: extracting data type identifiers, data content category identifiers, field sensitivity identifiers, data association scope identifiers, and data aggregation degree identifiers from data attribute information; extracting compliance level identifiers, privacy constraint identifiers, de-identification rule identifiers, circulation restriction identifiers, and processing restriction identifiers from compliance attribute information; performing sensitivity level mapping based on data type identifiers, field sensitivity identifiers, data association scope identifiers, data aggregation degree identifiers, and compliance level identifiers to generate sensitivity level identifiers; and generating de-identification requirement identifiers based on privacy constraint identifiers and de-identification rule identifiers; extracting business scenario identifiers, business purpose identifiers, and business responsibility entity identifiers from business attribute information; and extracting access subject identifiers and role identifiers from permission attribute information. The system generates color-coded permission identifiers, access scope identifiers, and operation permission identifiers. It extracts flow restriction identifiers and processing restriction identifiers from compliance attribute information, generates authorized use identifiers based on business purpose identifiers, generates access permission boundary identifiers based on access subject identifiers, role permission identifiers, access scope identifiers, and operation permission identifiers, generates permitted flow identifiers based on business scenario identifiers, flow restriction identifiers, and business responsibility entity identifiers, and generates processing constraint identifiers based on processing restriction identifiers and business purpose identifiers. It also extracts generation time identifiers, archiving time identifiers, destruction time identifiers, and retention period identifiers from lifecycle attribute information, and generates retention period identifiers based on these identifiers. Sensitivity level identifier, authorized use identifier, access permission boundary identifier, permitted flow identifier, processing constraint identifier, de-identification requirement identifier, and retention period identifier are merged into regulatory constraint information in the regulatory shadow object. The regulatory constraint information is used to limit the compliance regulatory boundaries of the real data object in the entire process of processing. The sensitivity level identifier is used to limit the regulatory intensity, the authorized use identifier is used to limit the permitted use purpose, the access permission boundary identifier is used to limit the scope of permitted processing entities, the permitted flow identifier is used to limit the permitted target flow, the processing constraint identifier is used to limit the permitted processing method, the de-identification requirement identifier is used to limit the de-identification status before export or sharing, and the retention period identifier limits the life cycle status. The regulatory identifier, real data object identifier, data source identifier, current regulatory status identifier, and regulatory constraint information are written into the regulatory shadow object structure template to generate a regulatory shadow object. The regulatory shadow object is bound to the real data object through the one-to-one correspondence between the regulatory identifier and the real data object identifier, carries the regulatory requirements corresponding to the real data object through the regulatory constraint information, and records the regulatory status of the real data object in the process of collection, cleaning, fusion, storage, retrieval, processing, sharing, export, archiving, and destruction through the current regulatory status identifier.

[0023] In this embodiment, the generation of the real data behavior trajectory sequence specifically includes: The event fields of the entire process are parsed to extract the event occurrence time, processing node identifier, processing subject identifier, processing action type, input data identifier, output data identifier, and target flow identifier; The process involves uniformly processing the event occurrence time to generate an event time code, and performing category mapping on the processing node identifier, processing subject identifier, processing action type, input data identifier, output data identifier, and target flow direction identifier to generate processing node code, processing subject code, processing action code, input data code, output data code, and target flow direction code. The uniform time granularity processing includes converting the year, month, day, hour, minute, and second of the event occurrence time into a unified time format, and performing time alignment, time truncation, and time granularity merging processing according to a preset time window. The category mapping processing includes establishing a category mapping table corresponding to the processing node identifier, processing subject identifier, processing action type, input data identifier, output data identifier, and target flow direction identifier, assigning corresponding category numbers to each type of identifier according to the category mapping table, and converting each type of identifier into its corresponding category code. Based on event time encoding, processing node encoding, processing subject encoding, processing action encoding, input data encoding, output data encoding, and target flow direction encoding, a single event structured encoding unit is constructed. The single event structured encoding unit is used to represent the processing time, processing node, processing subject, processing action, input object, output object, and target flow direction of a real data object in a full-process processing event. Multiple single-event structured coding units corresponding to the same real data object are sequentially ordered according to the event occurrence time. Data flow connection relationships between sequentially related single-event structured coding units are established based on input data identifiers, output data identifiers, and target flow direction identifiers. The sequence of real data behavior trajectories is composed of multiple sequentially ordered single-event structured coding units and the data flow connection relationships between sequentially related single-event structured coding units. The data flow connection relationships include input-output connection relationships and flow direction connection relationships. The input-output connection relationship is formed by matching the input data identifier in the subsequent single-event structured coding unit with the output data identifier in the preceding single-event structured coding unit. The flow direction connection relationship is formed by matching the target flow direction identifier in the preceding single-event structured coding unit with the processing node identifier in the subsequent single-event structured coding unit.

[0024] In this embodiment, the generation of the regulatory shadow state transition chain specifically includes: Based on the processing action types in the sequence of real data behavior trajectories, the regulatory status type to be migrated corresponding to the regulatory shadow object is determined. Then, based on the regulatory constraint information in the regulatory shadow object, compliance constraint verification is performed on the processing action type, processing entity identifier, processing node identifier, and target flow identifier to obtain the regulatory status migration node. The regulatory status migration node records the regulatory identifier, real data object identifier, pre-migration regulatory status identifier, post-migration regulatory status identifier, processing action type, processing node identifier, processing entity identifier, input data identifier, output data identifier, target flow identifier, and event occurrence time identifier. The regulatory status types to be migrated include: collection regulatory status, cleaning regulatory status, fusion regulatory status, storage regulatory status, retrieval regulatory status, processing regulatory status, sharing regulatory status, export regulatory status, archiving regulatory status, and destruction regulatory status. Multiple regulatory state transition nodes are sequentially sorted according to the event occurrence time in the real data behavior trajectory sequence, and the state succession relationship between the regulatory state transition nodes is established based on the input data identifier, output data identifier, and target flow identifier. The multiple regulatory state transition nodes after time-series sorting and the state succession relationships between the preceding and following regulatory state transition nodes are encapsulated in a chain to generate a regulatory shadow state transition chain corresponding to the real data behavior trajectory sequence.

[0025] In this embodiment, the construction of the expected compliance migration chain specifically includes: The process node identification, execution order, input requirements, output requirements, and transition conditions in the business process template are analyzed to determine the scope of process nodes in the expected compliance migration chain. Specifically, this involves: extracting business process identification, business scenario identification, and business processing stage identification corresponding to real data objects from the business process template; matching the corresponding set of process nodes based on these identifications; and extracting the process node identification, execution order, input requirements, output requirements, and transition conditions for each process node from the set. The node input requirements include allowed data types, allowed data sources, and allowed data states, while the node output requirements include allowed data types. The process node set is configured to determine the type, allowed data flow direction, and allowed data status of each node. Node transition conditions include node preconditions, node postconditions, and node jump conditions. The process nodes are sequentially ordered according to their execution order. Based on node input requirements, it is determined whether the actual data object meets the corresponding process node's data input conditions. Based on node output requirements, it is determined whether the actual data object meets the corresponding process node's data output conditions. Based on node transition conditions, it is determined whether adjacent process nodes meet the node transition conditions. Process nodes that simultaneously meet the data input, data output, and node transition conditions are retained. The node transition relationships between the retained process nodes are established according to the node execution order, generating the expected compliant migration chain's process node range. Based on regulatory constraint information in the regulatory rule base and regulatory shadow objects, constraints are imposed on the processing subject, processing action, data flow, processing method, de-identification status, lifecycle status, and output result status within the scope of process nodes to determine the compliance constraint boundaries of the expected compliance migration chain. Specifically, according to the execution order of nodes within the scope of process nodes, the processing subject, processing action, data flow, processing method, de-identification status, lifecycle status, and output result status corresponding to each process node are read; rule constraints corresponding to each process node are matched from the regulatory rule base, and the matched rule constraints are associated and mapped with the regulatory constraint information in the regulatory shadow objects; based on the association mapping results, the compliance constraint boundaries of each process node are determined. The system performs entity permission verification for processing entities, action compliance verification for processing actions, flow range verification for data flow, processing restriction verification for processing methods, de-identification requirement verification for de-identification status, retention period verification for lifecycle status, and output constraint verification for output result status. It retains process nodes that simultaneously satisfy entity permission verification, action compliance verification, flow range verification, processing restriction verification, de-identification requirement verification, retention period verification, and output constraint verification. Based on the processing entity, processing action, data flow, processing method, de-identification status, lifecycle status, and output result status corresponding to the retained process nodes, it establishes node constraint relationships to generate the compliance constraint boundaries of the expected compliance migration chain. Extract historical compliance regulatory status sequences from historical compliance processing samples, and encode the historical processing nodes, historical processing subjects, historical processing actions, historical data flow, historical processing methods, historical desensitization status, historical life cycle status, and historical output result status in the historical compliance regulatory status sequences to generate historical compliance regulatory status feature sequences. Specifically, following the order of occurrence in the historical compliance and regulatory status sequence, each historical compliance and regulatory status is sequentially read, including the historical processing node, historical processing subject, historical processing action, historical data flow, historical processing method, historical desensitization status, historical lifecycle status, and historical output result status. For each historical processing node, a corresponding node category number and node sequence position number are matched; for each historical processing subject, a corresponding subject category number, role category number, and permission level number are matched; for each historical processing action, a corresponding action category number and action sequence number are matched; for each historical data flow, a corresponding flow direction number and flow target number are matched; for each historical processing method, a corresponding processing category number and processing constraint level number are matched; for each historical desensitization status, a corresponding desensitization status number and desensitization intensity number are matched; and for each historical lifecycle status, a corresponding historical processing node, role category, role category, and permission level number are matched. The system matches the lifecycle stage number and lifecycle duration number corresponding to the current state, and matches the output result category number and output risk level number corresponding to the historical output result state. The system concatenates the node category number, node sequence position number, subject category number, role category number, permission level number, action category number, action sequence number, flow direction number, flow target number, processing category number, processing constraint level number, desensitization status number, desensitization intensity number, lifecycle stage number, lifecycle duration number, output result category number, and output risk level number corresponding to the same historical compliance and regulatory state in a fixed field order to generate a single-state feature encoding unit. Multiple single-state feature encoding units are arranged according to the state occurrence order in the historical compliance and regulatory state sequence to generate a historical compliance and regulatory state feature sequence. The historical compliance and regulatory status feature sequence is input into the Informer model. Temporal feature extraction and regulatory status association modeling are performed through time location encoding layer, encoder layer, probabilistic sparse attention layer and decoder layer to predict the compliance benchmark sequence corresponding to the real data object. Specifically, the process involves: inputting each individual state feature encoding unit from the historical compliance regulatory status feature sequence into the temporal position encoding layer of the Informer model according to the state occurrence order; overlaying the corresponding state sequence position information and time interval position information onto each individual state feature encoding unit to generate a position encoding feature sequence containing state temporal relationships; inputting the position encoding feature sequence into the encoder layer; and extracting regulatory status association features from the historical processing node change relationship, historical processing action change relationship, historical data flow change relationship, historical processing method change relationship, historical desensitization status change relationship, historical lifecycle status change relationship, and historical output result status change relationship through multi-layer temporal feature mapping to generate a regulatory status association feature sequence; and inputting the regulatory status association feature sequence into the probabilistic sparse attention layer to calculate the regulatory status relationship. The key regulatory state association feature sequence is generated by analyzing the state association strength between individual state feature encoding units in the joint feature sequence and selecting key regulatory state association relationships with state association strengths higher than a preset association threshold. The state association strength is calculated by taking the state temporal distance difference between individual state feature encoding units, the number of matching numbers between different categories, and the consistency of state change directions. A weighted association calculation is then performed on the state temporal distance difference, the number of matching numbers, and the consistency of change directions. The key regulatory state association feature sequence is input into the decoder layer, and state mapping and reconstruction are performed on the state association relationships in the key regulatory state association feature sequence according to the order of regulatory state changes. This generates a compliance regulatory state sequence corresponding to the real data object, and outputs a compliance benchmark sequence corresponding to the real data object based on the compliance regulatory state sequence. The compliance benchmark sequence is decoded to obtain the benchmark processing node, benchmark processing subject, benchmark processing action, benchmark data flow, benchmark processing method, benchmark desensitization status, benchmark life cycle status and benchmark output result status. Within the scope of process nodes and compliance constraint boundaries, the decoded benchmark statuses are chained together according to the status sequence to construct the expected compliance migration chain corresponding to the real data object. Status decoding includes: reading the category number information of each category in the compliance benchmark sequence one by one according to the status arrangement order in the compliance benchmark sequence, and performing category restoration, status mapping and field association processing on each category number information based on the preset number mapping relationship to generate the corresponding benchmark status; The chain combination includes: arranging each baseline state in sequence according to the state time sequence, establishing a state transition relationship based on the previous baseline data flow and the subsequent baseline processing node, establishing a state succession relationship based on the previous baseline output result state and the subsequent baseline processing action, and connecting multiple baseline states according to the state transition relationship and the state succession relationship to generate the expected compliance migration chain.

[0026] In this embodiment, the generation of the regulatory offset feature set specifically includes: Using the real data object identifier, regulatory identifier, and state occurrence order as a unified alignment benchmark, an inter-chain node index relationship is established for the regulatory shadow state migration chain and the expected compliance migration chain. Specifically, the state occurrence order of the shadow chain is determined according to the event occurrence time identifier in the regulatory shadow state migration chain, and the state occurrence order of the compliance chain is determined according to the state time sequence in the expected compliance migration chain. Regulatory state migration nodes with the same real data object identifier and corresponding state occurrence order are paired with expected compliance migration nodes, and the regulatory identifier is written into the pairing result to form an inter-chain node index relationship. Based on the inter-chain node index relationship, the processing node identifier in the regulatory status migration node is aligned with the expected processing node in the expected compliance migration chain at the node level to obtain the node-level alignment result. Specifically, the processing node identifier in the regulatory shadow status migration chain and the expected processing node in the corresponding expected compliance migration node are read sequentially according to the order of state occurrence. It is then determined whether the processing node identifier, node flow relationship, and node arrangement order are consistent with the expected processing node, expected node flow relationship, and expected node arrangement order. When the corresponding results are consistent, a node matching relationship is established; when the corresponding results are inconsistent, the node offset position is recorded. The node matching relationship and the node offset position are merged to generate the node-level alignment result. Under the node-level alignment results, the processing action type, target flow identifier, and actual use identifier are matched with the expected processing action, expected data flow, and permitted use constraints, respectively. Specifically, according to the node correspondence in the node-level alignment results, the processing action type, target flow identifier, and processing subject identifier in the regulatory status migration node, as well as the expected processing action, expected data flow, and permitted use constraints in the corresponding expected compliance migration node, are read sequentially. It is then determined whether the processing action type is consistent with the expected processing action and whether the target flow identifier is consistent with the expected data flow. Based on the processing action type, processing subject identifier, and target flow identifier, the actual use identifier is determined, and it is determined whether the actual use identifier is within the permitted use constraints. When the corresponding results are consistent, a corresponding matching relationship is established; when the corresponding results are inconsistent, the corresponding offset position is recorded. The corresponding matching relationship and the corresponding offset position are merged to generate action-level alignment results, flow-level alignment results, and use-level alignment results. Based on the access permission boundary identifier, de-identification requirement identifier, and retention period identifier, the processing subject identifier, de-identification status, and lifecycle status are matched with the expected processing subject, expected de-identification status, and expected lifecycle status, respectively. Specifically, according to the node correspondence in the node-level alignment results, the processing subject identifier, de-identification status, and lifecycle status in the regulatory status migration node, and the expected processing subject, expected de-identification status, and expected lifecycle status in the corresponding expected compliance migration node are read sequentially. It is determined whether the processing subject identifier is consistent with the expected processing subject and is within the scope of the access permission boundary identifier, whether the de-identification status is consistent with the expected de-identification status and meets the de-identification rules corresponding to the de-identification requirement identifier, and whether the lifecycle status is consistent with the expected lifecycle status and is within the scope of the retention period identifier. When the corresponding results meet the constraints, a corresponding matching relationship is established; when the corresponding results do not meet the constraints, the corresponding offset position is recorded. The corresponding matching relationships and corresponding offset positions are merged to generate the access permission level alignment result, the de-identification level alignment result, and the lifecycle level alignment result. Based on the output data identifier, target flow identifier, and post-migration regulatory status identifier, the output result status in the regulatory shadow status migration chain is aligned with the expected output result status at the output result level. Specifically, according to the node correspondence in the node-level alignment result, the output data identifier, target flow identifier, and post-migration regulatory status identifier in the regulatory shadow status migration chain, as well as the expected output result status in the corresponding expected compliance migration node, are read sequentially. It is then determined whether the output data type, target flow, and output status are consistent with the corresponding content of the expected output result status. When the corresponding results are consistent, an output result matching relationship is established; when the corresponding results are inconsistent, the output result offset position is recorded. The output result matching relationship and the output result offset position are merged to generate an output result-level alignment result. Based on the alignment results at the node, action, flow, purpose, permission, anonymization, lifecycle, and output levels, the offset fields, offset order, and offset range of the regulatory shadow state migration chain relative to the expected compliance migration chain are determined. Specifically, the offset positions in the node, action, flow, purpose, permission, anonymization, lifecycle, and output level alignment results are read, and the identifiers of inconsistent processing nodes, processing action types, target flow identifiers, actual purpose identifiers, processing subject identifiers, anonymization status, lifecycle status, and output status are extracted to generate offset fields. The regulatory state migration nodes corresponding to each offset field are read according to the order of state occurrence to determine the order of appearance of each offset field in the regulatory shadow state migration chain, and the offset order is generated based on the node flow relationship between the offset fields. Based on the starting and ending regulatory state migration nodes corresponding to the offset fields, the intervals of nodes with consecutive offsets are determined, and the offset range is generated based on the number of nodes with consecutive offsets, the node flow path, and the target flow range. The offset fields, offset order, and offset range are respectively merged into the permission offset feature, usage offset feature, flow offset feature, processing offset feature, desensitization offset feature, lifecycle offset feature, and output result offset feature to generate a regulatory offset feature set; Specifically, the following steps are taken: First, the offset fields, offset order, and offset range corresponding to the processing subject identifier being inconsistent with the expected processing subject or exceeding the scope of the access permission boundary identifier are grouped into permission offset features. Second, the offset fields, offset order, and offset range corresponding to the actual use identifier exceeding the permitted use constraints are grouped into use offset features. Third, the offset fields, offset order, and offset range corresponding to the target flow identifier being inconsistent with the expected data flow are grouped into flow offset features. Fourth, the offset fields, offset order, and offset range corresponding to the processing action type involving processing actions being inconsistent with the expected processing action are grouped into processing offset features. Fifth, the offset fields, offset order, and offset range corresponding to the desensitization rules corresponding to the desensitization status being inconsistent with the expected desensitization status or not meeting the desensitization requirements are grouped into desensitization offset features. Sixth, the offset fields, offset order, and offset range corresponding to the lifecycle status being inconsistent with the expected lifecycle status or exceeding the scope of the retention period identifier are grouped into lifecycle offset features. Seventh, the offset fields, offset order, and offset range corresponding to the output data type, target flow, or output status being inconsistent with the content corresponding to the expected output result status are grouped into output result offset features, thus generating a regulatory offset feature set.

[0027] In this implementation method, the generation of big data full-process supervision results and supervision and handling strategies specifically includes: Based on the offset field, offset order and offset range in the regulatory offset feature set, the corresponding regulatory state migration node is located, the processing node identifier, processing subject identifier, processing action type and target flow identifier are extracted, and regulatory state migration nodes with the same offset field and continuous offset range are merged to generate offset nodes. Based on the categories of permission offset features, purpose offset features, flow offset features, processing offset features, desensitization offset features, lifecycle offset features, and output result offset features, determine the offset type corresponding to the offset node; Based on the number of offset fields within the same offset node, the constraint strength and offset range corresponding to the regulatory constraint information, the offset amplitude corresponding to the offset node is determined. Specifically, the number of inconsistent offset fields within the same offset node is counted, the constraint strength of the regulatory constraint information corresponding to each offset field is read, the number of regulatory state migration nodes covered by the offset field within the offset range is determined, and the number of offset fields, constraint strength and number of covered nodes are weighted and merged to generate the offset amplitude corresponding to the offset node. The position of each offset node in the regulatory shadow state migration chain is determined according to the offset order. The first offset node is determined as the offset starting position, and the offset influence range is determined according to the offset range. The offset node, offset type, offset magnitude, offset starting position, and offset impact range are associated and encapsulated with the real data object identifier, regulatory identifier, and target flow identifier to generate big data full-process regulatory results. Based on the offset type and offset magnitude in the big data full-process supervision results, the corresponding supervision and handling conditions are matched, and the corresponding supervision and handling strategies are generated based on the matched supervision and handling conditions. The supervision and handling strategies include permission convergence strategy, sharing restriction strategy, de-identification enhancement strategy, abnormal export blocking strategy, data rollback strategy, lifecycle compression strategy, and audit enhancement strategy. The permission convergence strategy is a regulatory action strategy for revoking, downgrading, or blocking access permissions, data access permissions, data export permissions, and data sharing permissions that exceed the limits of the access permission boundary identifier. It is generated as follows: read the processing subject identifier, processing action type, target flow identifier, and offset magnitude corresponding to the permission offset feature; determine the access subject to be dealt with based on the processing subject identifier; determine the permission category to be dealt with based on the processing action type; determine the flow object to be dealt with based on the target flow identifier; and determine the permission revocation strength, permission downgrade strength, or permission blocking strength according to the offset magnitude to generate the permission convergence strategy. The sharing restriction policy is a regulatory action strategy that restricts, suspends, or prohibits data sharing behavior that exceeds the permitted flow direction identifier or permitted use constraint. It is generated as follows: the target flow direction identifier in the flow direction offset feature is filtered by flow direction intersection with the permitted flow direction identifier, and the actual use identifier in the use offset feature is filtered by use compliance with the permitted use constraint. The sharing path that meets both the flow direction intersection screening and the use compliance screening is retained, the sharing path that fails the screening is deleted, and the sharing restriction policy is generated based on the deletion result. The data masking enhancement strategy is a regulatory action strategy that supplements data masking, increases the masking level, or prohibits plaintext output for data processing results that do not meet the masking requirements and the corresponding masking rules. Its generation method is as follows: the masking status in the masking offset feature is compared with the masking requirement identifier to determine the masking gap. This gap is then mapped to masking actions such as supplementary masking, field generalization, field deletion, or ciphertext replacement. These actions are combined according to the output scenario corresponding to the target flow identifier to generate the data masking enhancement strategy. The masking gap is the difference between the masking status in the masking offset feature and the corresponding masking rule of the masking requirement identifier. This difference includes unmasked fields, fields with insufficient masking level, fields with mismatched masking methods, and plaintext output fields. The abnormal export blocking strategy is a regulatory handling strategy that blocks, suspends, or transfers data export behavior that does not conform to the expected output result status in terms of output data type, target flow direction, or output status. The generation method is as follows: the output data type, target flow direction, and output status in the output result offset characteristics are checked against the expected output result status. If any check fails, the corresponding output data identifier is locked, the corresponding export task is frozen, and an export blocking, export suspension, or manual review instruction is generated based on the item that failed the check, thus forming the abnormal export blocking strategy. The data rollback strategy is a regulatory handling strategy for data processing results that involve processing actions and are inconsistent with the expected processing actions, which involves version rollback, result cancellation, or link restoration. Its generation method is as follows: locate the regulatory state migration node where the processing offset occurs based on the processing offset characteristics, trace the previous input data identifier and the subsequent output data identifier of the regulatory state migration node, determine the data version generated by the abnormal processing, replace the data version generated by the abnormal processing with the data version corresponding to the previous input data identifier, restore the corresponding regulatory state, and generate the data rollback strategy. The lifecycle compression strategy is a regulatory disposal strategy for data whose lifecycle status exceeds the retention period identifier limit or is inconsistent with the expected lifecycle status, such as early archiving, shortening the retention period, or triggering destruction. The generation method is as follows: read the lifecycle status and retention period identifier according to the lifecycle offset characteristics, calculate the degree of overdue status of the current lifecycle status relative to the retention period identifier, map the degree of overdue status to the early archiving period, the compressed retention period, or the destruction trigger time, and generate the lifecycle compression strategy. The audit enhancement strategy is a regulatory action strategy for regulatory anomalies that are triggered by multiple offset features simultaneously or whose offset magnitude reaches the audit enhancement conditions. This strategy involves encrypting and retaining logs, increasing the audit frequency, or triggering manual review. The strategy is generated by: statistically analyzing the number of offset types, offset nodes, and offset impact range in the set of regulatory offset features; constructing an audit tracking object set; writing the regulatory status migration nodes, processing entity identifiers, processing action types, and target flow identifiers in the audit tracking object set into the enhanced audit queue; and generating the audit enhancement strategy that triggers log encryption and retention, increased audit frequency, or manual review.

[0028] A big data-driven end-to-end monitoring system based on intelligent analytics includes: The event acquisition module is used to acquire a set of information on events and regulatory attributes throughout the entire process. The shadow construction module is used to construct regulatory shadow objects that correspond to real data objects; The trajectory generation module is used to generate real data behavior trajectory sequences for the entire process of structured coding events. The state transition module is used to drive the regulatory shadow object to perform regulatory state transitions and generate a regulatory shadow state transition chain. The compliance building module is used to build the expected compliance migration chain corresponding to the real data objects; The offset analysis module is used to perform shadow chain offset alignment processing and generate a set of regulatory offset features. The results output module is used to determine offset information, generate big data full-process supervision results, and output supervision and handling strategies.

[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to a data full-process supervision scenario of a provincial government big data platform. This government big data platform is deployed in a data center in East China, operating over multiple consecutive business cycles. The platform involves various types of real data objects, including population information, enterprise information, approval information, and public service information. Different business departments access and transfer data through data sharing interfaces, data processing links, and external collaboration interfaces. Due to frequent cross-departmental data sharing, a large number of processing entities, and complex data flows within the platform, traditional supervision methods based on log auditing and static rule matching are insufficient to promptly detect hidden data deviations during sharing, processing, and export, easily leading to problems such as usage deviations, flow deviations, and abnormal desensitization states.

[0030] In this scenario, the system first acquires the entire process of real data objects during collection, cleaning, fusion, storage, retrieval, processing, sharing, export, archiving, and destruction, and simultaneously obtains the corresponding regulatory attribute information. Then, based on the regulatory attribute information, the system constructs a regulatory shadow object, generates a sequence of real data behavior trajectories by structurally encoding the entire process events, and drives the regulatory shadow object's execution state migration according to the processing action type and regulatory constraint information, forming a regulatory shadow state migration chain. Simultaneously, the system combines business process templates, regulatory rule bases, and historical compliance regulatory state sequences to construct an expected compliance migration chain, and identifies permission offset characteristics, usage offset characteristics, flow offset characteristics, and desensitization offset characteristics through shadow chain offset alignment processing. When abnormal cross-departmental sharing behavior is detected in some real data objects, the system can locate the corresponding regulatory state migration node, identify the abnormal target flow identifier and the output state of the identifier that does not meet the desensitization requirements, and automatically generate sharing restriction policies and desensitization enhancement policies to implement restriction processing on the abnormal sharing link.

[0031] During continuous operation, the system conducts regulatory analysis on large-scale end-to-end processing events generated by multiple business departments, and both the regulatory shadow state migration chain and the expected compliance migration chain remain stably constructed. For data processing links exhibiting abnormal flow behavior, the system can identify abnormal export states before data export and execute export freeze processing through abnormal export blocking strategies. For data objects with abnormal lifecycles, the system can identify data lifecycle states that have exceeded the retention period and trigger lifecycle compression strategies. Through these processing methods, abnormal data flow behaviors in the platform are effectively identified, the ability to track regulatory links and locate anomalies is improved, and the compliance, security, and efficiency of regulatory handling of the big data platform's end-to-end supervision are enhanced.

[0032] Table 1. Performance Comparison of the Invention and Traditional Big Data Full-Process Supervision Methods

[0033] As can be clearly seen from Table 1, the method of the present invention is superior to the traditional method in many indicators.

[0034] The abnormal behavior identification rate of the method of this invention is 92.6%, while that of traditional regulatory methods is 89.3%. This improvement mainly comes from the shadow chain offset alignment processing between the regulatory shadow state migration chain and the expected compliance migration chain. The system can continuously analyze the state migration process of real data objects, rather than making judgments based on a single log, a single node, or static rules. Therefore, it is more stable in identifying abnormal behaviors across nodes and processing links.

[0035] Regarding the detection rate of data flow anomalies, the method of this invention achieves 88.9%, while the traditional regulatory method achieves 84.7%. Regarding the detection rate of data anonymization anomalies, the method of this invention achieves 90.1%, while the traditional regulatory method achieves 86.4%. This invention incorporates target flow identifiers, anonymization status, permitted flow identifiers, and anonymization requirement identifiers into regulatory offset analysis. By aligning at the flow level and anonymization level, it identifies the differences between the actual processing status and the expected compliance status, thus uncovering inconsistencies in flow offset and anonymization status that are easily overlooked by traditional regulatory methods.

[0036] Regarding the abnormal export identification rate, the method of this invention achieves 89.4%, while the traditional regulatory method achieves 85.2%. This improvement stems from the fact that this invention comprehensively judges whether the output data type, target flow direction, and output status match the expected output status during output result alignment, and generates an abnormal export blocking strategy upon detecting anomalies. Therefore, this invention can not only identify the export action itself, but also combine the export object, export flow direction, and export status to determine whether the export behavior is compliant, thereby improving the accuracy of identifying abnormal export behavior.

[0037] In terms of processing efficiency, the generation latency of the regulatory results using the method of this invention is 376ms, compared to 418ms for traditional regulatory methods; the response time for anomaly handling is 481ms, compared to 536ms for traditional regulatory methods. This improvement stems from the fact that the regulatory shadow object carries the regulatory identifier, the current regulatory status identifier, and regulatory constraint information. The system can perform state transition analysis, offset identification, and handling strategy matching without directly reading the real data content, thus reducing the overhead of data parsing and redundant verification.

[0038] Regarding the accuracy of data flow tracking, the method of this invention achieves 92.3%, while the traditional regulatory method achieves 88.1%; regarding the false alarm rate, the method of this invention achieves 5.8%, while the traditional regulatory method achieves 7.1%. This invention uses a structured encoding of event occurrence time, input data identifier, output data identifier, and target flow identifier through real data behavior trajectory sequences, and establishes a continuous state succession relationship by combining it with the regulatory shadow state migration chain, thus providing regulatory judgments with more complete contextual basis.

[0039] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A big data-driven end-to-end monitoring method based on intelligent analysis, characterized in that: Includes the following steps: Acquire the full-process processing events of real data objects in the big data platform and the corresponding regulatory attribute information set; Construct regulatory shadow objects that correspond to real data objects based on a set of regulatory attribute information; The entire process of events is structured and encoded according to the event occurrence time, processing node identifier, processing subject identifier, processing action type, input data identifier, output data identifier, and target flow direction identifier to generate a sequence of real data behavior trajectories. Based on the processing action types in the real data behavior trajectory sequence and the regulatory constraint information in the regulatory shadow object, the regulatory shadow object is driven to perform regulatory state transition and generate a regulatory shadow state transition chain. Based on business process templates and regulatory constraint information, the scope of process nodes and the boundaries of compliance constraints are determined. The historical compliance and regulatory status sequence is feature-encoded, and a compliance benchmark sequence is generated through the Informer model to construct the expected compliance migration chain. Perform shadow chain offset alignment processing on the regulatory shadow state migration chain and the expected compliance migration chain to generate a regulatory offset feature set; Based on the set of regulatory offset features, offset information is determined, big data full-process regulatory results are generated, and corresponding regulatory handling strategies are output.

2. The big data full-process supervision method based on intelligent analysis according to claim 1, characterized in that, The full-process processing events refer to the processing events generated during the collection, cleaning, fusion, storage, retrieval, processing, sharing, export, archiving, and destruction of real data objects.

3. The big data full-process supervision method based on intelligent analysis according to claim 1, characterized in that, The construction of the regulatory shadow object specifically includes: The data attribute information, business attribute information, permission attribute information, compliance attribute information, and lifecycle attribute information in the regulatory attribute information set are processed to standardize fields, unify identifiers, and normalize timestamps to generate a standardized set of regulatory attributes. Based on the data attribute information and business attribute information in the standardized regulatory attribute set, generate real data object identifiers, data source identifiers and regulatory identifiers, and establish a one-to-one correspondence between regulatory identifiers and real data object identifiers; A regulatory shadow object structure template is constructed based on a standardized set of regulatory attributes. The regulatory shadow object structure template does not set a field for storing real data content, but is only used to carry the regulatory identifier, real data object identifier, data source identifier, current regulatory status identifier and regulatory constraint information corresponding to the real data object. Based on the data attribute information and compliance attribute information in the standardized regulatory attribute set, determine the sensitivity level identifier and de-identification requirement identifier corresponding to the real data object; based on the business attribute information, permission attribute information and compliance attribute information, determine the authorized use identifier, access permission boundary identifier, permitted flow identifier and processing constraint identifier corresponding to the real data object; based on the life cycle attribute information, determine the retention period identifier corresponding to the real data object. The sensitivity level identifier, authorized use identifier, access permission boundary identifier, permitted flow identifier, processing constraint identifier, desensitization requirement identifier, and retention period identifier are merged into the regulatory constraint information in the regulatory shadow object; Write the regulatory identifier, real data object identifier, data source identifier, current regulatory status identifier, and regulatory constraint information into the regulatory shadow object structure template to generate a regulatory shadow object.

4. The big data full-process supervision method based on intelligent analysis according to claim 1, characterized in that, The generation of the real data behavior trajectory sequence specifically includes: The event fields of the entire process are parsed to extract the event occurrence time, processing node identifier, processing subject identifier, processing action type, input data identifier, output data identifier, and target flow identifier; The event occurrence time is uniformly processed at the time granularity to generate an event time code. Then, category mapping processing is performed on the processing node identifier, processing subject identifier, processing action type, input data identifier, output data identifier, and target flow direction identifier to generate processing node code, processing subject code, processing action code, input data code, output data code, and target flow direction code. A single-event structured coding unit is constructed based on event time coding, processing node coding, processing subject coding, processing action coding, input data coding, output data coding, and target flow coding; Multiple single-event structured coding units corresponding to the same real data object are sequentially ordered according to the time of event occurrence. Data flow connection relationships between sequentially related single-event structured coding units are established based on input data identifiers, output data identifiers, and target flow direction identifiers. The sequence of real data behavior trajectories is composed of multiple sequentially ordered single-event structured coding units and the data flow connection relationships between sequentially related single-event structured coding units.

5. The big data full-process supervision method based on intelligent analysis according to claim 1, characterized in that, The generation of the regulatory shadow state transition chain specifically includes: Based on the processing action type in the real data behavior trajectory sequence, determine the regulatory status type to be migrated corresponding to the regulatory shadow object, and perform compliance constraint verification on the processing action type, processing subject identifier, processing node identifier and target flow identifier based on the regulatory constraint information in the regulatory shadow object to obtain the regulatory status migration node; Multiple regulatory state transition nodes are sequentially sorted according to the event occurrence time in the real data behavior trajectory sequence, and the state succession relationship between the regulatory state transition nodes is established based on the input data identifier, output data identifier, and target flow identifier. The multiple regulatory state transition nodes after time-series sorting and the state succession relationships between the preceding and following regulatory state transition nodes are encapsulated in a chain to generate a regulatory shadow state transition chain corresponding to the real data behavior trajectory sequence.

6. The big data full-process supervision method based on intelligent analysis according to claim 1, characterized in that, The construction of the expected compliance migration chain specifically includes: Analyze the process node identifiers, node execution order, node input requirements, node output requirements, and node transition conditions in the business process template to determine the scope of process nodes in the expected compliance migration chain; Based on the regulatory constraint information in the regulatory rule base and regulatory shadow objects, constraints are imposed on the processing subject, processing action, data flow, processing method, desensitization status, life cycle status and output result status in the scope of process nodes to determine the compliance constraint boundaries of the expected compliance migration chain. Extract historical compliance regulatory status sequences from historical compliance processing samples, and encode the historical processing nodes, historical processing subjects, historical processing actions, historical data flow, historical processing methods, historical desensitization status, historical life cycle status, and historical output result status in the historical compliance regulatory status sequences to generate historical compliance regulatory status feature sequences. The historical compliance and regulatory status feature sequence is input into the Informer model. Temporal feature extraction and regulatory status association modeling are performed through time location encoding layer, encoder layer, probabilistic sparse attention layer and decoder layer to predict the compliance benchmark sequence corresponding to the real data object. The compliance benchmark sequence is decoded to obtain the benchmark processing node, benchmark processing subject, benchmark processing action, benchmark data flow, benchmark processing method, benchmark desensitization status, benchmark life cycle status, and benchmark output result status. Within the scope of process nodes and compliance constraint boundaries, the decoded benchmark statuses are chained together according to the status sequence to construct the expected compliance migration chain corresponding to the real data object.

7. The big data full-process supervision method based on intelligent analysis according to claim 1, characterized in that, The generation of the regulatory offset feature set specifically includes: Using real data object identifiers, regulatory identifiers, and the order of status occurrence as a unified alignment benchmark, an inter-chain node index relationship is established for the regulatory shadow status migration chain and the expected compliance migration chain. Based on the inter-chain node index relationship, the processing node identifier in the regulatory status migration node is aligned with the expected processing node in the expected compliance migration chain at the node level to obtain the node-level alignment result. Under the node-level alignment results, the processing action type, target flow direction identifier, and actual use identifier are matched with the expected processing action, expected data flow direction, and permitted use constraints, respectively. Based on the access permission boundary identifier, de-identification requirement identifier, and retention period identifier, the processing subject identifier, de-identification status, and lifecycle status are matched with the expected processing subject, expected de-identification status, and expected lifecycle status, respectively. Based on the output data identifier, target flow identifier, and post-migration regulatory status identifier, the output result status in the regulatory shadow status migration chain is aligned with the expected output result status at the output result level. Based on the alignment results at the node level, action level, flow level, purpose level, permission level, desensitization level, lifecycle level, and output result level, determine the offset fields, offset order, and offset range of the regulatory shadow state migration chain relative to the expected compliance migration chain. The offset fields, offset order, and offset range are respectively merged into the permission offset feature, purpose offset feature, flow offset feature, processing offset feature, desensitization offset feature, lifecycle offset feature, and output result offset feature to generate a regulatory offset feature set.

8. The big data full-process supervision method based on intelligent analysis according to claim 1, characterized in that, The generation of the big data full-process supervision results and supervision and handling strategies specifically includes: Based on the offset field, offset order and offset range in the regulatory offset feature set, the corresponding regulatory state migration node is located, the processing node identifier, processing subject identifier, processing action type and target flow identifier are extracted, and regulatory state migration nodes with the same offset field and continuous offset range are merged to generate offset nodes. Based on the categories of permission offset features, purpose offset features, flow offset features, processing offset features, desensitization offset features, lifecycle offset features, and output result offset features, determine the offset type corresponding to the offset node; The offset magnitude corresponding to the offset node is determined based on the number of offset fields within the same offset node, the constraint strength corresponding to the regulatory constraint information, and the offset range. The position of each offset node in the regulatory shadow state migration chain is determined according to the offset order. The first offset node is determined as the offset starting position, and the offset influence range is determined according to the offset range. The offset node, offset type, offset magnitude, offset starting position, and offset impact range are associated and encapsulated with the real data object identifier, regulatory identifier, and target flow identifier to generate big data full-process regulatory results. Based on the offset type and offset magnitude in the big data full-process supervision results, corresponding regulatory handling conditions are matched, and corresponding regulatory handling strategies are generated based on the matched regulatory handling conditions. The regulatory handling strategies include permission convergence strategy, sharing restriction strategy, de-identification enhancement strategy, anomaly export blocking strategy, data rollback strategy, lifecycle compression strategy, and audit enhancement strategy.

9. A big data end-to-end monitoring system based on intelligent analysis, comprising executing the big data end-to-end monitoring method based on intelligent analysis as described in any one of claims 1 to 8, characterized in that, include: The event acquisition module is used to acquire a set of information on events and regulatory attributes throughout the entire process. The shadow construction module is used to construct regulatory shadow objects that correspond to real data objects; The trajectory generation module is used to generate real data behavior trajectory sequences for the entire process of structured coding events. The state transition module is used to drive the regulatory shadow object to perform regulatory state transitions and generate a regulatory shadow state transition chain. The compliance building module is used to build the expected compliance migration chain corresponding to the real data objects; The offset analysis module is used to perform shadow chain offset alignment processing and generate a set of regulatory offset features. The results output module is used to determine offset information, generate big data full-process supervision results, and output supervision and handling strategies.

Images