Cross-system data non-inductive acquisition and governance fusion method in field of internal control supervision

Abstract

The invention relates to a cross-system data non-inductive acquisition and governance fusion method in the field of internal control supervision, which comprises the following steps of: performing data acquisition from a production data source of each business system through a low-intrusive or non-intrusive technical approach, and acquiring modes including database transaction log analysis, RPA, multi-modal file automatic extraction and application interface adaptation; data standardization and cleaning, supervision subject domain division, core entity analysis and global ID generation are carried out on an accessed original data stream, and data which is treated and marked with global ID tags are organized according to supervision subject domains and are stored in a unified supervision data resource library; a standard data service API is provided for the internal control supervision system, and event data flow is supervised. The problem that an existing data supply technology cannot provide efficient and high-quality data support for the internal control supervision system is solved, the stability and the real-time performance are improved, and the data use complexity and the development cost of the internal control supervision system are greatly reduced.

Description

Technical Field

[0001] This invention relates to a cross-technology of internal control supervision and enterprise data governance, specifically to a method and system that provides data support for internal control supervision systems and enables seamless data collection, intelligent governance and integration across multiple independent business systems. Background Technology

[0002] The core function of an internal control and oversight system is to identify and warn of hidden violations and risks based on data from various internal and external business systems (such as project management systems, contract monitoring systems, financial management systems, and logistics support systems), combined with oversight needs, and to implement a closed-loop approach to handling identified risks. Efficiently, accurately, and comprehensively acquiring data from these business systems is fundamental to the effectiveness of the constructed internal control and oversight system. However, because each business system is often planned independently during its construction, data access methods and coding standards vary, posing significant challenges to data collection and subsequent integration and utilization.

[0003] Currently, the closest existing technologies for providing data to internal control and supervision systems mainly include the following two: 1. Batch synchronization method based on data platform / data warehouse This is currently the most common data integration solution. This solution uses ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform) tools to perform batch data synchronization during idle periods in business systems (such as at night), extracting data, either fully or incrementally, from the production databases of various business systems to a centralized enterprise-level data platform or data warehouse. The internal control and supervision system, as the data consumer, then retrieves data from this centralized data source for regulatory analysis.

[0004] 2. Business systems directly provide API interfaces. This approach involves each business system developing a dedicated data query interface for the internal control and supervision system. When the supervision system needs data, it retrieves specific business data in real time by calling these APIs.

[0005] Although the above technologies achieve physical data centralization or interface connectivity, they all have the following significant drawbacks and reasons: Disadvantage 1: The problem of data "logical silos" is prominent, and the degree of automation in cross-system correlation supervision is low.

[0006] Reasons: In Method 1, the data infrastructure primarily achieves "physical centralization" of data, but fails to address the issue of deep logical data integration. The data models and coding rules of different systems remain independent, requiring the internal control and monitoring system to manually write complex, customized relational query scripts and attempt to correlate data using methods such as fuzzy matching. This process cannot be automated and suffers from poor accuracy. In Method 2, the internal control and monitoring system needs to call multiple APIs separately and resolve data correlations manually, resulting in a complex architecture and low efficiency.

[0007] Disadvantage 2: The data collection process is either intrusive or lacks timeliness, affecting the real-time nature and stability of supervision.

[0008] Reasons: Method 1's batch synchronization feature results in a long data update cycle (usually T+1), which cannot meet the real-time monitoring requirements for important business changes. Method 2 requires customized development of the business system, which is intrusive; at the same time, frequent API calls may put pressure on the performance of the business system, and its stability is greatly affected by changes in the business system.

[0009] Disadvantage 3: Data quality governance is lagging behind, and the internal control and supervision system itself bears a heavy burden.

[0010] Reason: The data provided by the two methods mentioned above is mostly "raw material" or "semi-finished product" for regulatory scenarios. Before use, internal control and regulatory systems must consume significant computing resources and development effort to perform secondary data cleaning, transformation, and quality improvement. This essentially shifts the complexity of data governance to the application layer, resulting in a bloated internal control and regulatory system architecture and severely restricting the development and operational efficiency of core regulatory logic.

[0011] In summary, the core challenge of existing related technologies lies in their failure to provide an end-to-end solution that combines seamless data collection, intelligent governance, and deep integration capabilities, and can directly output "regulatory-ready" data products.

[0012] This application aims to address the problem that existing technologies cannot provide efficient and high-quality data support for internal control and supervision systems. Specifically, it requires solving the following three interrelated technical challenges: 1. Challenges of data collection in terms of intrusiveness, timeliness, and comprehensiveness: How to achieve real-time, automated, and comprehensive collection of data from various heterogeneous data sources within an enterprise (including databases, legacy systems, unstructured files, etc.) without modifying business systems or affecting their performance.

[0013] 2. The challenge of deep integration of cross-system data: How to intelligently manage the collected multi-source heterogeneous data, solve the problems of inconsistent formats, varying quality, and different encodings, and make a core breakthrough in the problem of chaotic identification of cross-system business entities, so as to achieve deep integration and unified identification of data at the logical level.

[0014] 3. The challenge of "regulatory readiness" for data products: How to encapsulate the governed data into data assets with unified semantics, strong correlation, high quality, and direct usability by the internal control and supervision system, thereby freeing them from complex data preprocessing work and allowing them to focus on precise cross-system supervision. Summary of the Invention

[0015] To address the issue that existing data supply technologies cannot provide efficient and high-quality data support for internal control and supervision systems, a cross-system, seamless data collection and governance integration method is proposed in the field of internal control and supervision. The core of this method lies in constructing an automated data processing pipeline covering data collection, governance, and service. Its core architecture is as follows: Figure 1 As shown, it includes a core data processing layer and a data service layer.

[0016] The technical solution of this invention is as follows: A cross-system data seamless collection and governance integration method in the field of internal control and supervision includes: Step S1, Multi-Source Heterogeneous Data Access and Seamless Acquisition: This step involves acquiring data from production data sources across various business systems using low-intrusion or non-intrusive techniques. Depending on the technical conditions of each business system, appropriate acquisition methods are selected, including database transaction log parsing, Robotic Process Automation (RPA, a technology that uses software robots to simulate user operations to collect data), automatic multimodal file extraction, and application interface adaptation, to achieve the acquisition of core business data and its peripheral semi-structured and unstructured data. For any data source, a full synchronization is performed upon initial access to obtain a baseline state. Subsequently, based on the technical characteristics of each acquisition method, the system automatically switches to the corresponding incremental synchronization or periodic polling mode to continuously capture data changes.

[0017] Step S2: Intelligent Governance and Integration for Internal Control Supervision S2.1: Perform data standardization and cleaning on the raw data stream accessed by S1; S2.2: Regulatory Subject Domain Division, Core Entity Resolution, and Global ID Generation: Regulatory subject area division: From the perspective of internal control supervision, establish regulatory subject areas; Core Entity Parsing and Global ID Generation: Utilizing graph algorithms, fuzzy string matching, and semantic analysis techniques from natural language processing, entity identifiers from different systems are correlated and matched to eliminate ambiguity in the identification of the same entity in different systems. A global regulatory object ID is assigned to the unique business entity identified, thereby logically integrating all its associated data into a unified entity profile, achieving deep data fusion at the entity level. S2.3: Construct a unified regulatory data resource repository: Organize the data that has been governed and tagged with a global ID according to the regulatory subject domain and store it in the unified regulatory data resource repository; at this point, the data in the repository is a data network that is deeply integrated across systems with the "regulatory object" as the center; Step S3: Standardized data supply for the internal control and supervision system: Provide a standard data service API: encapsulate a set of RESTful APIs to provide query services to the internal control and supervision system; Provide regulatory event data stream: Establish a real-time message queue, and when the status of a core regulatory entity changes, proactively push the event to the internal control and supervision system to drive it to perform real-time rule analysis.

[0018] Furthermore, the specific data acquisition methods in S1 include: Database transaction log parsing: This is the preferred collection method for capturing changes to structured business data in real time at the database level; by configuring and enabling the database log parsing tool, the database transaction log can be read and parsed directly. RPA: For business systems that cannot directly connect to the database or lack effective data interfaces, robotic process automation technology is used for data collection. RPA automatically logs into the business system, navigates to a specific data query page, executes query operations, and accurately captures and parses structured data from the system's graphical user interface by simulating the operations of real users. Automatic extraction of multimodal files: By monitoring the file server or the file directory specified by the business system, key business files are automatically obtained. By comprehensively using technologies including OCR optical character recognition, NLP natural language processing and document structure parsing, key structured data fields are automatically identified and extracted from files of various formats including PDF, Word, Excel and images, so as to realize the automated and intelligent acquisition of file form data. Application Interface Adaptation: For business systems with existing data service interfaces, data collection is performed by calling their provided read-only APIs.

[0019] Furthermore, data standardization and cleaning in S2.1 includes: format conversion, illegal value cleaning, null value handling, and code unification.

[0020] The beneficial effects of this invention are as follows: 1. Achieved truly seamless cross-system supervision: Through "supervisory subject domain division" and "core entity parsing," a global supervisory object ID was generated, enabling data that was originally scattered across different systems to be automatically associated and deeply integrated around business entities. This allows the internal control and supervision system to directly perform cross-system, integrated analysis and monitoring of "the entire project lifecycle," "the entire contract execution process," and "the entire fund flow chain," solving the technical problem of poor data correlation.

[0021] 2. Achieved seamless data collection from business systems, improving stability and real-time performance: By comprehensively employing technologies such as database transaction log parsing, RPA, and automatic multimodal file extraction, comprehensive seamless data collection was achieved from the technical and operational levels to the data format level. Database log parsing avoided code modification and performance impact on business systems, achieving zero-intrusion and near real-time capture of core data; RPA technology solved the data access issues of legacy and closed systems; and multimodal extraction technology enabled the transformation of unstructured files. Together, these technologies constructed a stable, real-time, and comprehensive seamless data collection system.

[0022] 3. Significantly reduces the data complexity and development cost of the internal control and supervision system: This solution outputs "regulatory-ready" data that has undergone deep governance, unified semantics, and strong correlation. The internal control and supervision system no longer needs to perform complex and time-consuming data preprocessing. It can be developed directly based on a clear data model and standard APIs, simplifying the system architecture and improving the response speed and operational efficiency of regulatory operations. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the system architecture and data processing of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0025] Figure 1 This invention presents the overall system architecture and data processing flowchart of the cross-system data seamless collection and governance fusion method in the field of internal control and supervision. The flowchart shows the flow and value-added process of regulatory data from the original business system to the final internal control and supervision application from bottom to top.

[0026] A method for seamless cross-system data collection and governance integration in the field of internal control and supervision includes the following steps: Step S1: Multi-source heterogeneous data access and seamless acquisition (corresponding to...) Figure 1 (Multi-source heterogeneous data access and seamless acquisition) from the data source layer to the core data processing layer This step involves collecting data from production data sources of various business systems (such as project management systems, contract monitoring systems, financial management systems, and logistics support systems) through a variety of low-intrusion or non-intrusion technical approaches. Based on the different technical conditions of the business systems, the optimal method is selected to collect core business data and its peripheral semi-structured and unstructured data.

[0027] Data Acquisition Method 1: Database Transaction Log Parsing: This is the preferred acquisition method, used to capture changes in structured business data in real time at the database level. By configuring and enabling database log parsing tools, it directly reads and parses the database's transaction logs. This method can capture all data insertion, update, and delete operations and output them in the form of a data change event stream. It has zero intrusion into the business system itself, does not consume application server resources, and can achieve real-time data capture at the millisecond to second level.

[0028] Data Collection Method 2: Robotic Process Automation (RPA, a technology that uses software robots to simulate user operations to collect data): For business systems that cannot directly connect to a database or lack effective data interfaces, RPA is used for data collection. RPA automatically logs into the business system, navigates to a specific data query page, executes the query, and accurately extracts and parses structured data from the system's graphical user interface by simulating real user operations. This process is treated as normal user operations by the business system, requiring no modification or development, thus achieving seamless data collection at the operational level.

[0029] Data Acquisition Method 3: Automatic Extraction of Multimodal Files: This method aims to solve the challenge of collecting critical business data stored in unstructured files. By monitoring file servers or specified file directories in business systems, it automatically acquires critical business files (such as contracts, payment vouchers, and acceptance reports). It comprehensively utilizes technologies such as OCR (Optical Character Recognition), NLP (Natural Language Processing), and document structure parsing to automatically identify and extract key structured data fields (such as contract number, amount, date, and supplier name) from files in various formats including PDF, Word, Excel, and images, achieving automated and intelligent acquisition of file-based data.

[0030] Data collection method 4: Application interface adaptation: For business systems with existing data service interfaces, data is collected by calling their provided read-only APIs.

[0031] For any data source, upon initial access, a full synchronization is performed using any one of the applicable methods 1-4 above to obtain the baseline status. Subsequently, based on the technical characteristics of each acquisition method, the system automatically switches to the corresponding incremental synchronization or timed polling mode to continuously capture data changes.

[0032] Step S2: Intelligent governance and integration for internal control and supervision (corresponding to) Figure 1 The core data processing layer's internal workflow includes: "Data Standardization and Cleaning" → "Core Entity Resolution and Global ID Generation" → "Unified Regulatory Data Resource Repository". This step represents a qualitative change in data, transforming it from "raw" to "regulatory ready."

[0033] S2.1: Data Standardization and Cleaning: Formatting, cleaning illegal values, handling null values, and unifying codes for the raw data stream accessed by S1.

[0034] S2.2: Regulatory Subject Domain Division, Core Entity Resolution, and Global ID Generation: Regulatory subject area division: From the perspective of internal control supervision, establish regulatory subject areas including "projects", "contracts", "units", "assets", "cash flow" and "equipment".

[0035] Core entity parsing and global ID generation: This is crucial for achieving cross-system data fusion. Utilizing graph algorithms, fuzzy string matching, and semantic analysis from natural language processing, entity identifiers (such as project numbers, contract numbers, and unified social credit codes) from different systems are correlated and matched. This eliminates ambiguity in the identification of the same entity across different systems. A unique global regulatory object ID is assigned to each identified business entity, logically integrating all associated data into a unified entity profile, thus achieving deep data fusion at the entity level.

[0036] S2.3: Construct a unified regulatory data resource repository: Organize the data that has been governed and tagged with a global ID according to the regulatory subject domain, and store it in a unified regulatory data resource repository. At this point, the data in the repository has become a data network centered on the "regulated object" and deeply integrated across systems.

[0037] Step S3: Standardized data supply for the internal control and supervision system (corresponding to...) Figure 1 (Data Service Layer → Data Application Layer) Supply 1: Provide a standard data service API: Encapsulate a set of RESTful APIs to provide query services to the internal control and supervision system. For example, based on the global project ID, retrieve the full and related data of the project in all relevant systems.

[0038] Supply 2: Provide regulatory event data stream: Establish a real-time message queue, and when the status of a core regulatory entity changes, proactively push the event to the internal control and supervision system to drive it to perform real-time rule analysis.

[0039] Alternative solutions, for example: In the data acquisition phase, for systems that do not support database log parsing, their standard data interfaces (APIs) can be used for functionally equivalent access; for systems that have neither logs nor open interfaces, RPA technology can be used to collect data from the application interface.

[0040] In the entity parsing stage, the core entity parsing method implemented using graph algorithms, fuzzy string matching, and semantic analysis techniques from natural language processing is only a preferred implementation. In practical applications, precise matching based on predefined rules or machine learning models trained on historical data can also be used for discrimination. As long as they can achieve unified identification and association of business entities across systems, they are all equivalent substitutes for the technical solution of this invention.

[0041] In the data supply stage, APIs and event streams are merely examples of standardized supply; their specific protocols and formats can be adjusted according to the technology stack of the actual regulatory system.

[0042] The above-described embodiments are merely one implementation of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention should be determined by the appended claims.

Claims

1. A cross-system data non-sensing collection and governance fusion method in the field of internal control supervision, characterized in that, Comprise: Step S1, multi-source heterogeneous data access and non-sensing collection: this step collects data from the production data sources of various business systems through low-invasive or non-invasive technical approaches; according to the different technical conditions of the business system, the corresponding collection mode is selected, including database transaction log analysis, robot process automation RPA, multi-modal file automatic extraction, application interface adaptation, to realize the collection of core business data and its peripheral semi-structured and unstructured data; for any data source, a full synchronization is performed at the first access to obtain the baseline state, and then according to the technical characteristics of each collection mode, automatically switch to the corresponding incremental synchronization or timed polling mode to continuously capture data changes; Step S2, intelligent governance and fusion for internal control supervision: S2.1: data standardization and cleaning of the original data stream accessed by S1; S2.2: supervision theme domain division, core entity analysis and global ID generation: Supervision theme domain division: from the perspective of internal control supervision, establish a supervision theme domain; Core entity analysis and global ID generation: use techniques including graph algorithms, fuzzy string matching, and semantic analysis in natural language processing to associate and match entity identifiers from different systems, eliminate the identification ambiguity of the same entity in different systems, and assign a global supervision object ID to the unique business entity identified, and then logically integrate all associated data of the entity into a unified entity portrait, realizing the deep fusion of data at the entity level; S2.3: build a unified supervision data repository: organize and store the data that has been governed and tagged with a global ID in a unified supervision data repository according to the supervision theme domain; at this time, the data in the repository is a data network centered on "supervision objects" and deeply integrated across systems; Step S3, standardized data supply for internal control supervision system: Provide standard data service API: encapsulate a set of RESTful API to provide query services to the internal control supervision system; Provide supervision event data stream: establish a real-time message queue and actively push events to the internal control supervision system when the status of the core supervision entity changes to drive real-time rule research and judgment.

2. The cross-system data non-sensing collection and governance fusion method in the internal control supervision field according to claim 1, characterized in that, The collection mode in S1 specifically includes: Database transaction log analysis: this is the preferred collection mode, used to capture structured business data changes in real time from the database level; by configuring and enabling the database log analysis tool, the transaction log of the database is directly read and analyzed; RPA: for business systems that cannot be directly connected to the database or lack effective data interfaces, use robot process automation technology for collection; RPA simulates the operation of a real user, automatically logs in to the business system, navigates to the specific data query page, performs query operations, and accurately captures and analyzes structured data from the system graphical user interface; Multi-modal file automatic extraction: By monitoring the file server or the specified file directory of the business system, automatically obtain the key business files, and comprehensively use the technologies including OCR optical character recognition, NLP natural language processing and document structure analysis to automatically identify and extract the key structured data fields from the files in multiple formats including PDF, Word, Excel and pictures, and realize the automatic and intelligent acquisition of file form data. Application interface adaptation: For the business system with existing data service interface, data collection is carried out by calling the read-only API provided by the system.

3. The cross-system data non-sensing collection and governance fusion method in the internal control supervision field according to claim 1, characterized in that, The data standardization and cleaning in S2.1 include: format conversion, illegal value cleaning, null value processing and code unification.

Images