Data matching method and device based on safety approval, equipment and medium

Abstract

The invention relates to the technical field of data processing, can be applied to business scenes such as financial science and technology, medical treatment and health, and discloses a data matching method based on security approval, which comprises the following steps: creating a data table and configuring access authority, inserting data in the table into an acquisition table and then synchronizing to an intermediate database and adding a field identifier; synchronizing the data of the intermediate database to a distributed storage structure partitioned according to dates through a timed task; receiving a data table name, a data type and identification information, determining a matching field based on the data type, matching the data value in the distributed storage with a target field, and generating a result set; and submitting a matching result as a security signature, triggering an approval process, and sending result data to the terminal according to permission mapping after approval is passed. According to the method, the data processing efficiency is improved and the sensitive data circulation risk is reduced by constructing the structured acquisition process, setting the automatic synchronization mechanism, executing data matching based on the field type and integrating the approval control logic.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data matching method, device, equipment and storage medium based on security approval. Background Art

[0002] In the field of financial technology business, business personnel often need to obtain corresponding core attribute information for a batch of existing identification data (such as policy number, customer number, agent number, etc.) in their daily work, such as insurance structure, customer risk label, agent sales situation, etc., to assist in subsequent business evaluation, compliance review or risk analysis. This type of data query and extraction demand initiated based on identification is usually classified as "matching data demand". However, the existing data service platforms are mostly based on query interfaces or reports, which are difficult to support processing modes such as batch input of identification fields, automatic association of system base tables, and batch generation of target data. Especially in scenarios involving customer-sensitive fields, due to restrictions on authority control and security policies, existing systems usually desensitize or restrict the display of sensitive information at the interface level, making manual queries inefficient and incomplete.

[0003] In the healthcare business field, there are similar demands for matching data acquisition. For example, operations personnel often need to batch organize medical records corresponding to a set of patient identifiers (such as ID number, mobile phone number, and medical card number) for use in risk stratification, patient return visits, and medical insurance compliance inspections. However, most existing information systems provide information presentation centered around a single patient, lacking the automated ability to batch-drive the extraction of associated fields and control output based on input identifiers. For queries involving patient privacy information, security approval and other operations must also be coordinated, further lengthening the overall response time and affecting business agility.

[0004] In current enterprise data management systems, the acquisition of matching data often relies on manual processing. Business personnel typically export the data to be matched and then enlist the assistance of the development team to complete querying, integration, and cleansing. This is especially true when fields from multiple source systems, such as customers, policies, and agents, require developers to write separate SQL statements for processing. This approach not only results in long response times but also creates a significant duplication of development resources, becoming a major bottleneck in effectively utilizing data in practice. Furthermore, because data is distributed across different systems and has inconsistent table structures, the manual integration process is prone to problems such as misunderstanding of fields and information omissions, further reducing the reliability of data quality.

[0005] Furthermore, in terms of data security management, matching requirements generally involve customer privacy fields or key business information, and the acquisition process must undergo a rigorous approval process. Existing technologies often separate the approval process from the data extraction process, failing to automatically integrate user permissions, table authorization, and business content for coordinated control. This results in the approval process being easily disconnected from data distribution, increasing compliance risks and increasing the operational burden on data management departments. Especially in the context of the growing scale and multi-scenario nature of matching requirements, the current model that relies on "manual + approval + development" can no longer effectively support the compliance, efficiency, and security requirements of batch sensitive data extraction.

[0006] To sum up, existing technologies for automatic extraction of matching data have problems such as insufficient batch processing capabilities, inability to securely obtain sensitive fields, disconnection between approval and distribution processes, and waste of developer resources. There is an urgent need for a technical solution that can support user self-service operations, has secure approval linkage capabilities, and can automatically complete multi-table matching and field extraction to improve overall processing efficiency and data governance compliance. Summary of the Invention

[0007] The main purpose of the present invention is to provide a data matching method, device, equipment and storage medium based on security approval, aiming to solve the technical problems of inaccurate sensitive data matching, poor process security and slow response speed caused by decentralized authority, low efficiency of cross-system data association and delayed manual approval in the manual matching data acquisition method in the existing technology.

[0008] To achieve the above objectives, the present invention provides a data matching method based on security approval, comprising:

[0009] Create a user data table and set access authorization for the user data table;

[0010] Identify the authorized user data table through the permission table, and insert the data in the user data table into the collection data table;

[0011] Setting a scheduled task to synchronize the data in the collected data table to an intermediate database, and adding a source table name field and a synchronization date field to the intermediate database;

[0012] Synchronize the data of the intermediate database to the date partition table of distributed storage, and establish partition storage for the date partition table based on the synchronization date field;

[0013] Receive input content of the user identification field, the target data table name field, and the data type field on the report page;

[0014] Determine the data table associated field according to the data type field, match the data value field of the date partition table with the data table associated field based on the target data table name field, and generate a matching result set;

[0015] Submitting a security sign-off including the user identification field and the matching result set through a report portal, and executing a security approval process based on the security sign-off;

[0016] When the security approval process is completed, the matching result set is sent to the user terminal according to the binding relationship between the user identification field in the authority table and the name field of the target data table.

[0017] Furthermore, to achieve the above-mentioned purpose, the present invention provides a data matching device based on security approval, comprising:

[0018] A user data table configuration module is used to create a user data table and set access authorization for the user data table;

[0019] A data collection and processing module is used to identify the authorized user data table through the permission table and insert the data in the user data table into the collection data table;

[0020] An intermediate data synchronization module is used to set a scheduled task to synchronize the data in the collected data table to the intermediate database, and to add a source table name field and a synchronization date field to the intermediate database;

[0021] A distributed storage management module, configured to synchronize the data of the intermediate database to a date partition table of distributed storage, and to establish partition storage for the date partition table based on the synchronization date field;

[0022] The report input receiving module is used to receive the input content of the user identification field, the target data table name field and the data type field on the report page;

[0023] a data matching execution module, configured to determine a data table associated field according to the data type field, match the data value field of the date partition table with the data table associated field based on the target data table name field, and generate a matching result set;

[0024] A security sign-on approval module, configured to submit a security sign-on containing the user identification field and the matching result set through a report portal, and execute a security approval process based on the security sign-on;

[0025] The matching result sending module is used to send the matching result set to the user terminal according to the binding relationship between the user identification field in the authority table and the name field of the target data table when the security approval process is completed.

[0026] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a data matching program based on security approval stored in the memory and executable on the processor. When the data matching program based on security approval is executed by the processor, the steps of the data matching method based on security approval as described above are implemented.

[0027] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data matching program based on security approval is stored. When the data matching program based on security approval is executed by a processor, the steps of the data matching method based on security approval as described above are implemented.

[0028] Beneficial effects: The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. It discloses a data matching method based on security approval, including: creating a data table and configuring access rights, inserting the data in the table into the collection table and synchronizing it to the intermediate database and adding field identifiers, synchronizing the intermediate database data to a distributed storage structure partitioned by date through a scheduled task; receiving the data table name, data type and identification information, determining the matching field based on the data type, matching the data value in the distributed storage with the target field and generating a result set; submitting the matching result as a security report and triggering the approval process, and sending the result data to the terminal according to the permission mapping after approval. The present invention realizes batch extraction, classification matching and permission compliance delivery of different types of data identification information by constructing a structured collection process, setting an automatic synchronization mechanism, performing data matching based on field type, and integrating approval control logic, thereby improving data processing efficiency and reducing the risk of sensitive data circulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0030] Figure 1 A schematic diagram of an application environment of a data matching method based on security approval in an embodiment of the present invention;

[0031] Figure 2 This is a flow chart of an embodiment of a data matching method based on security approval according to the present invention;

[0032] Figure 3 This is a functional module diagram of a preferred embodiment of a data matching device based on security approval according to the present invention;

[0033] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0034] Figure 5FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] The data matching method based on security approval provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can create a data table and configure access rights through the user end, insert the data in the table into the collection table, synchronize it to the intermediate database and add field identifiers, and synchronize the intermediate database data to a distributed storage structure partitioned by date through a scheduled task; receive the data table name, data type and identification information, determine the matching field based on the data type, match the data value in the distributed storage with the target field and generate a result set; submit the matching result as a security report and trigger the approval process, and after approval, send the result data to the terminal according to the permission mapping. The present invention realizes batch extraction, classification matching and permission compliance delivery of different types of data identification information by constructing a structured collection process, setting an automatic synchronization mechanism, performing data matching based on field type, and integrating approval control logic, thereby improving data processing efficiency and reducing the risk of sensitive data circulation. Among them, the user end can be but is not limited to various personal computers, laptops, smart phones, tablets and portable wearable devices. The server end can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0037] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a data matching method based on security approval provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0038] like Figure 2 As shown, the data matching method based on security approval proposed by the present invention includes the following steps:

[0039] S10, creating a user data table and setting access authorization for the user data table;

[0040] In this embodiment, creating a user data table requires a set of structured field information as a foundation. These fields include a data sequence number field, a data type field, a data value field, and a user identification field. The data sequence number field is used to establish data order control and is typically defined as an integer field, marking the physical sequence number of the imported data. The data type field is used to indicate the matching type represented by each data item. This field is limited to character types, and the value range is controlled by a system-defined enumeration set, such as "policy number," "customer number," and "agent number." The data value field is the key field value involved in the actual matching process. The data type is a variable-length character type and allows the storage of various identification information, such as ID numbers and mobile phone numbers. The user identification field records the identity of the data submitter or data owner, serving as an identifier for access control and supporting user-level isolation. After the fields are defined, the system dynamically constructs a table creation statement based on the preset field order and generates a database hierarchy based on the field names and types. The generated data table naming rules combine the user ID and timestamp to generate a unique identifier to avoid naming conflicts.

[0041] After completing the creation of the data table structure, the system binds the name of the table to the user identification field and registers it in the permission table to form the basic metadata for permission verification. The permission table records the mapping relationship between each data table accessible to the collection module and its authorized accessor. Before the collection end performs automatic collection, it is necessary to confirm whether the target data table has an authorization relationship through the permission table comparison. The system automatically grants read permissions to the collection user account, and at the same time grants the account write permissions to the target collection data table to complete the data transfer path configuration. The access credential number is generated by the system based on the mapping between the user ID and the authorization table, and is a unique identifier. After generation, the credential information is returned to the front-end user and written to the access control module as an access verification token to ensure that data access requests can only be triggered by subjects with valid credentials.

[0042] In an environment where the database platform supports automatic table creation, after receiving information containing field definitions, the backend service can dynamically construct the table creation SQL. This table creation instruction is not generated through a template, but is mapped to the database field type table based on the field data type to complete type binding. For example, the data sequence field is automatically parsed as INT type, the data type field is mapped to VARCHAR(32), the data value field is set to VARCHAR(256), and the user identifier field is set to VARCHAR(64). The field order cannot be adjusted. Before constructing the statement, the field order legitimacy can be checked through a field comparison algorithm. If there is an abnormal field order, the system will block the table creation process and generate a structure abnormality log.

[0043] When generating table names, a hash function is used to encrypt the user ID field and then concatenate the result with a timestamp. For example, SHA-1 is used to encrypt the user ID and then concatenate it, such as 20250406, to generate a name like cjpp_auto_6f1d2a20250406. This prevents user and table name conflicts and ensures traceability. The system configuration allows you to configure whether the naming rules support customized extension fields, such as business line codes or task identifier prefixes. After naming is completed, the system writes the mapping between the table name and the user ID field into the permissions table. This permissions table is an independent metadata management table that contains the fields table name, user identification number, table creation time, and validity status.

[0044] To automate access authorization, the system uses native database permission management statements to grant data read permissions. For example, if the collection user is the system account collector_user, permissions are granted via GRANT SELECT ON cjpp_auto_xxx TO collector_user, and write permissions are granted simultaneously via GRANT INSERT ON target_collect_table TO collector_user. The generated access credential number is a 16-digit alphanumeric combination that can be generated using UUID clipping and stored in the credential control table. The control table records the credential number, user identification field, authorization scope, and expiration time, supporting time control and revocation operations. Authorization information is injected into the access control module via configuration, completing data protection at the service call level.

[0045] Example: In a healthcare business scenario, the hospital's data audit department needs to verify a batch of patient identification numbers submitted via an external interface to assess whether these patients have expired medical records in the core health record system. The hospital's data management team imports the ID numbers into a data table with a consistent structure. The system creates the table based on the configured field order and type, automatically generates access credentials, authorizes the collection program to access the table, synchronizes the data to an intermediate database, and performs backend matching. The data table created in this scenario is named with the hospital department identifier for subsequent tracking by business line.

[0046] In a fintech business scenario, the customer compliance analysis team needs to verify whether a batch of mobile phone numbers belongs to high-risk customers and analyze their historical account transactions. Business personnel import the phone numbers and risk categories into the platform. The platform creates a standard structured data table and immediately grants access to the data collection module. Permission control is achieved by binding access credentials to the permission table to achieve isolation, preventing non-task-related users from accessing sensitive phone numbers. Complete data creation immediately connects to the subsequent automatic matching process, reducing development involvement and improving compliance verification efficiency.

[0047] By uniformly constraining data table structures and field standards, while also introducing permission tables, access credential numbers, and database-level authorization mechanisms, we can configure controllable data access paths during table creation, preventing subsequent acquisition tasks from failing due to missing permissions. Data table names utilize an encrypted composite structure to ensure uniqueness and support audit traceability of the original user. Structured storage of permission relationship metadata supports rapid location and permission verification. This approach achieves complete control over data access security, table structure legitimacy verification, and authorization path closure, effectively ensuring standardization, automation, and traceability during the data preparation phase prior to large-scale matching data processing.

[0048] S20, identifying the authorized user data table through the permission table, and inserting the data in the user data table into the collection data table;

[0049] In this embodiment, the permission table refers to a data structure used to record table access authorization relationships in a database or data management system. It usually exists in the form of an independent table. The fields include user identification value, authorized data table name, authorization type (such as read permission), authorization status, creation time, etc., which are used to verify whether the user has the permission to access a certain data table during the data collection or data distribution stage. The process of identifying authorized user data tables refers to parsing the authorization records in the permission table and screening out a list of data table name fields with valid authorization status. This operation relies on the user identification value as the retrieval condition, which is usually bound to the current access session or credential number to ensure the accuracy and consistency of permission identification.

[0050] The user data table refers to a data carrier that has been standardized and authorized in the early creation process. It has a unified structure, consistent field order, and has fields such as data number fields and data value fields for subsequent matching processing. The operation of inserting data into the collection data table is an act of moving authorized data to the task general processing table. The purpose is to centrally process data from multiple sources, unify format verification, and standardize cleaning. The insertion process is based on field-level data structure conversion to ensure that the fields in the user data table are aligned with the predefined structure of the collection data table. After the structure is aligned, the data is written using a batch insert strategy to avoid the performance overhead brought by row-by-row insertion, and transaction control is used to prevent partial writes from causing inconsistent data status.

[0051] The collection data table is a standard data pool established in the system. Its field structure is fixed and its meaning is clear. It typically includes fields such as source table information, data value fields, collection time, and task number. This table is used for subsequent data flow and identification, providing data support for distributed transit. Fields must be mapped one-to-one to avoid ambiguous interpretations and provide support for subsequent data desensitization, cleansing, and audit log generation.

[0052] Throughout the entire process, permission judgment takes precedence, and data access must rely on permission confirmation results to be executed. Collection operations must be strictly restricted by the source authorization mechanism to ensure that the data source is legal, the operation is traceable, and the behavior is controllable.

[0053] Example: In the fintech sector, the operations support department of a large life insurance company is responsible for handling data requests for "streamlined information verification." These requests, often submitted by sales personnel, include customer identifiers across various dimensions, such as policy numbers, customer numbers, and mobile phone numbers. The goal is to quickly verify whether these identifiers exist in internal systems, whether they have risk tags, and whether they have triggered blacklist rules. Because many of these data fields contain sensitive customer information, the system implements a strict permission table mechanism to prevent unauthorized access and information leakage. Sales personnel's data entry tables must adhere to naming conventions and be pre-configured with user identifier values. The permission table records their table access authorization. Once the authorization is verified, the system automatically extracts data from the designated table and inserts it into the collected data table. This mechanism eliminates repeated permission verification processes and enables data to be collected in a unified format into the backend data cleansing area, enabling automated matching, desensitization, and risk tagging of batches of customer numbers. This process reduces manual data extraction costs and avoids unnecessary exposure of data before desensitization, meeting the dual requirements of rapid response and secure control.

[0054] In the healthcare sector, a regional medical management platform collects chronic disease screening data reported by primary healthcare facilities within its jurisdiction. Each facility is required to regularly upload a batch of resident ID numbers to verify the medical insurance system for hospitalization or outpatient records of chronic conditions such as hypertension and hyperlipidemia. Because each facility's data system varies and the uploaded table structure must be standardized, the system uses a pre-established permission table to restrict access to each facility's authorized data upload interface. After a primary healthcare facility uploads a data table, the system verifies that the facility has read access to the table and verifies that the table structure is compliant. Upon successful permission verification, the system automatically extracts the ID number fields from the data table and inserts them into a collection data table. The collection process also tags the table name and the uploading facility information, facilitating subsequent identity correlation analysis within the medical insurance big data platform. This unified permission control and collection mechanism not only avoids the inefficiency of manual table-by-table review but also ensures consistent structure of collected data, providing an accurate data foundation for subsequent machine processing and risk analysis.

[0055] By identifying and filtering the data tables accessible to the current subject through permission tables, data collection operations can be restricted to the authorized scope, preventing unauthorized access to data. Simultaneously, the introduction of data structure standardization, field-level format conversion, batch write mechanisms, and insertion logging effectively addresses issues such as heterogeneous data sources, inconsistent structures, performance bottlenecks, and recoverability after data insertion failures. This makes the collection process more automated, secure, and traceable, establishing a standardized closed-loop processing loop from data access to system collection.

[0056] S30, setting a scheduled task to synchronize the data in the collected data table to an intermediate database, and adding a source table name field and a synchronization date field to the intermediate database;

[0057] In this embodiment, the collected data table stores standardized data records aggregated from the user data table. To ensure that this data can be used by subsequent analysis modules or matching logic, it needs to be transferred from the current database system to an intermediate database with a more flexible and scalable structure for management. To achieve this goal, a periodically triggered data transfer mechanism is required. This mechanism is typically implemented through an operating system-level timer trigger or a scheduler function in the database. This mechanism can automatically execute data transfer scripts based on minute-level, hourly, or custom time intervals, and complete batch synchronization of data according to unified data extraction and insertion logic during each execution.

[0058] The starting point for data synchronization is the collection data table. During synchronization, the system batch extracts data records from the table and writes them one by one to the target data table in the intermediate database. This process is not just about transferring field values; it also includes metadata enhancement in two dimensions. The first is the writing of the source table name field. This field is used to identify the original source table to which the data originally belonged. It can be derived from the table name information saved during the previous collection, or it can be obtained by reading the permission configuration table or the collection record table before the synchronization is triggered. This field is a necessary auxiliary field in the intermediate database, enabling the tracing and differentiation of data sources in subsequent processing.

[0059] The second is the synchronization date field, which is used to record the time information of the current data synchronization. This field can use the system time when the task is executed, or it can be uniformly obtained from the current environment and injected into all inserted records before the script is executed. This field will serve as the basic metadata for subsequent construction of partition structures and execution of data lifecycle management. The field type is generally date or timestamp, and the specific format needs to be set according to the time type standard supported by the intermediate database. When adding this field, it is necessary to ensure the uniformity of the field format, and optionally enable the default value setting to handle edge cases such as write failure or field missing.

[0060] Field addition is typically completed during the SQL insert statement construction phase. The values ​​of the source table's name field and synchronization date field are dynamically concatenated as additional columns, written alongside the original data, to a structured table in the intermediate database. This structured table must have a predefined table structure that includes these two newly added fields and supports batch writes and high-frequency access.

[0061] Once the fields are written, each data record in the intermediate database possesses both content value and source information, improving data manageability and traceability. This structure enhances the structure during synchronization without changing the original collection table structure, providing foundational information support for subsequent operations such as data partitioning, filtering, matching, and approval.

[0062] In actual implementation, multiple methods can be used to synchronize data from the collected data table to the intermediate database and add the required field information. One method is to set timed trigger logic at the database layer using built-in scheduling functions (such as Oracle's DBMS_SCHEDULER, MySQL's EVENT mechanism, PostgreSQL's pgAgent, etc.). When the trigger time arrives, the scheduler executes the predefined SQL script to complete data extraction and target writing. For example, the following logic can be constructed: use the SELECT statement to read all or incremental data records from the collected data table; during the SQL statement construction phase, two fields are added: one is the statically populated source table name (this value can be obtained by passing parameters outside the synchronization script) and the other is the current system time (database time functions such as SYSDATE() can be used); use the INSERT INTO...SELECT... statement to write the data to the target table created in the intermediate database.

[0063] Another approach is to use middleware such as Apache Airflow, Spring Batch, or a custom data synchronization component at the data service layer, setting up a scheduler to execute scripts at specific times. Middleware can define task dependencies and execution logic through YAML configuration or Python task orchestration. Data extraction, transformation, field addition, and loading are automated through calls between components. The field addition process can be integrated into the data transformation function, centralizing data structure adjustments and metadata enhancement.

[0064] In addition, for scenarios with large data volumes, you can first export the collected data to an intermediate format (such as CSV or Parquet), then load it into an intermediate database. During the loading process, you can batch append the source table name and synchronization date field values. This approach is suitable for heterogeneous databases or cross-system data delivery.

[0065] In terms of field storage strategy, to ensure query efficiency, you can set the source table name field as the secondary index field of the intermediate database table, and set the synchronization date field as the partition key or as a query filter condition. This improves data retrieval speed and supports date-based cleanup of expired data.

[0066] By synchronizing the data in the collected data table to the intermediate database and adding the source table name field and synchronization date field during the writing process, the accuracy and efficiency of subsequent data management and calls can be significantly improved. On the one hand, the source table name field gives each record a clear source, which helps to accurately identify the source of data in multi-table concurrent synchronization or subsequent related analysis; on the other hand, the synchronization date field makes the data traceable over time, providing a basic basis for strategies such as data version management, partitioned storage, and expired cleanup. Overall, this design not only avoids modifications to the original collection table structure, but also builds an intermediate-tier data platform with high scalability, high controllability, and structural consistency, providing structural guarantees for subsequent field matching, permission filtering, and approval.

[0067] S40, synchronizing the data of the intermediate database to a date partition table of distributed storage, and establishing partition storage for the date partition table based on the synchronization date field;

[0068] In this embodiment, the data in the intermediate database needs to be synchronized regularly or in real time to a distributed storage platform with scalability and query performance advantages to enable efficient access and computational support for large-scale matching data for subsequent analysis tasks. To meet this requirement, a data synchronization operation must first be performed to migrate the records in the intermediate database to the target storage medium. During this process, the system must preserve the integrity of the structure fields and ensure that the data structure is consistent with the subsequent processing logic. The core fields of the synchronized data include the data value field, the user identification field, the data type field, the source table name field, and the synchronization date field. The synchronization date field will be used as the core partitioning basis in the subsequent construction of the partition storage structure.

[0069] To support efficient data access and partition management, a multi-level directory structure must be constructed during data synchronization based on the specific values ​​of the synchronization date field. This structure is typically organized into year, month, and day levels, with each level independently storing data for the corresponding date range. For example, if a synchronization record has a synchronization date of April 6, 2025, the system should store the corresponding data file in a partitioned directory with the path / 2025 / 04 / 06 / . The synchronization date field can be parsed as a timestamp and standardized to generate the directory structure, ensuring consistency across different data sources and time zone configurations.

[0070] For written file formats, use data structures that support columnar storage, high compression, and fast query capabilities. For example, mainstream formats such as Apache Parquet, ORC, or Delta Lake are recommended. These formats offer excellent compatibility and performance with big data query engines such as Hive, Spark, and Trino. When writing data, serialize structured records in the intermediate database into the target format, ensuring that field types do not lose precision or have semantic ambiguity during conversion.

[0071] When building partition directories, the system ensures partition path consistency and dynamically generates directory and subdirectory paths based on the synchronization date field value to prevent partition redundancy or overwriting. Directory names should avoid non-standard symbols and use zero-padding whenever possible to ensure lexicographical order is consistent with chronological order, thereby improving partition pruning efficiency in subsequent queries.

[0072] File writes typically utilize batch processing logic, processing a fixed number of intermediate database records at a time and generating independent data files within the partitioned path. After writing is complete, the system performs a file checksum to verify that the number of records written matches the synchronized number in the intermediate database. If there is a discrepancy, a rollback or alarm mechanism is implemented to ensure data integrity in distributed storage.

[0073] One specific implementation involves configuring a streaming or scheduled task flow on a data service platform (such as Apache NiFi, Flink, or Airflow), reading structured data from an intermediate database, and converting the data into a columnar structure using ParquetWriter or an equivalent serialization tool. Before writing, the system determines the target directory for the currently processed record based on the synchronization date field value. If the directory does not exist, a corresponding multi-level directory structure is dynamically created. The system writes data files in batches to the corresponding directories according to the configured batch write strategy and uses the atomic write mechanism provided by distributed file systems (such as HDFS, S3, and OSS) to ensure the integrity and consistency of the write operation. After writing is complete, a verification function is called to compare the actual number of records written in the current batch with the total number of records marked "pending synchronization" in the intermediate database. If there is a discrepancy, a rollback mechanism is activated, deleting the abnormal file and re-marking the synchronization status as failed. The system can also optionally append file metadata to record attributes such as the write time, write task number, and batch number to improve subsequent traceability.

[0074] Another implementation method is to use Spark Structured Streaming combined with the partitioned write mechanism to automatically project partitions on the synchronization date field. By specifying the partitionBy("sync_date") parameter during the DataFrame construction phase, Spark will automatically identify and generate directories at daily granularity when writing to HDFS or S3, supporting concurrent writes to different directories. Combined with DeltaLake or Hive table structure definitions, these partition paths can also be automatically registered at the metadata layer, facilitating direct partition pruning in subsequent SQL queries or metric calculations.

[0075] For enterprise environments with multi-center deployments across multiple locations, you can also design intermediate database synchronization operations as asynchronous message-driven, using message queues (such as Kafka and RocketMQ) to transmit synchronization tasks. The write service then subscribes to synchronization messages and pushes the corresponding data to the distributed storage directory structure. This approach is suitable for loosely coupled data bridging between heterogeneous systems.

[0076] By building a multi-level directory structure based on the synchronization date field during the data synchronization process and writing data files in a columnar storage format, the query performance and task processing efficiency of large-scale data can be effectively improved. On the one hand, the date partition structure allows the system to only scan files within the relevant date range when querying, significantly reducing IO costs; on the other hand, the introduction of columnar storage improves the compression ratio and concurrent read and write performance, providing a stable and efficient data foundation for subsequent matching operations, permission filtering, statistical analysis and other scenarios. At the same time, the synchronization date field, as a unique timestamp identifier, not only supports data partitioning, but also provides key support for functions such as data traceability, expiration cleanup, and version rollback, forming a flexible data structure for multi-source tasks.

[0077] S50, receiving input contents of a user identification field, a target data table name field, and a data type field on a report page;

[0078] In this embodiment, the report page serves as a user interaction interface and is the core entry point for the system to receive key parameters for analysis tasks. During the configuration process, three basic user parameters must be collected through a graphical interface or embedded input controls: user identification field, target data table name field, and data type field. The user identification field is used to identify the identity information of the request initiator and is the basic constraint condition for subsequent authority control and approval process of the matching results. The target data table name field corresponds to the data source table name that has previously completed authority binding, and is used to limit the range of matching data read by the system from distributed storage to prevent unauthorized access and data redundancy. The data type field is used to identify which semantic type field needs to be used as the matching basis for this matching operation, such as customer number, policy number, agent number, ID number, mobile phone number, etc., to determine the associated logical path between the subsequently selected fields and the basic fields of the system.

[0079] In order to ensure the validity of user-submitted information and system recognizability, the system needs to perform format restrictions and structural constraints on the input fields of the page before receiving the input content. For example, the user identification field can be set as a non-editable backfill field, which is automatically injected by the user when logged in to avoid the risk of forgery caused by manual filling. The target data table name field must support drop-down box selection or text entry and be formatted in combination with regular expressions to ensure that it complies with the preset naming strategy (such as starting with a specific prefix). The data type field can implement fixed enumeration value selection through a radio button or drop-down list. The enumeration value maintains a one-to-one correspondence with the matching field configured by the system to ensure that the system can correctly parse the subsequent field matching logic.

[0080] To support large-scale concurrent requests and configuration preservation, the system generates a structured query request model for the three fields entered by the user. This model provides a unified parameter context for subsequent data scheduling, matching, approval, recording, and distribution. The structured model consists of fields such as the original input field value, an automatically generated unique request number, a receipt timestamp, and a page source identifier. Once the input is complete, the server encapsulates the data and stores it in the configuration metadata repository, providing parameter support for multi-node asynchronous processing.

[0081] This input process does not directly trigger matching operations; it serves only as the starting point for subsequent data processing, ensuring that the system maintains sufficiently clear data boundaries and permission control conditions when performing matching, filtering, and security approval operations. Furthermore, this input action should be recorded as a request event, and the system can retain metadata corresponding to the page input content in the log system for auditing, tracing, or request reconstruction.

[0082] This input collection operation can be implemented using dynamic page components built on a Web framework. For example, a user ID field display box, a target data table name field input box, and a data type field drop-down box can be embedded in the front-end page built with Vue, React, or Angular, and interact with the backend API interface through a binding model. After the user logs in, the system automatically parses the current user's user ID and fills it into the page input box, and disables the field to prevent the user from manually modifying it. The target data table name field can support fuzzy matching and link the permission list, that is, based on the binding relationship of the current user in the permission table, the list of operable data table names is dynamically issued as candidate options. The data type field is configured with an enumeration input component based on the standard matching types currently supported by the platform, such as customer number, policy number, agent number, ID number, etc., to restrict users from selecting illegal data types.

[0083] After the input information is submitted, the front end can call a RESTfulAPI interface, and send a structured request model composed of three types of fields to the back-end server. After receiving the model, the back-end server automatically adds the current timestamp, page call identifier, and request unique number, and stores the structure in the configuration table or delivers it to the message queue for subsequent data scheduling system consumption. In order to prevent invalid requests from polluting the log system or triggering redundant processing, the system needs to perform a basic check on whether the target data table name field exists in the permission table and whether the data type field is a legal value after receiving the input. Requests that fail the check will be terminated and an error message will be returned to the front-end page. This implementation method can be deployed on the browser interface of the PC, or embedded in the integrated control of the enterprise's internal business system to ensure that business personnel can complete request input and task initialization operations without understanding the underlying database structure.

[0084] By receiving input from user identification fields, target data table name fields, and data type fields, the system can establish clear data processing boundaries, permission context, and matching semantic configurations, ensuring that subsequent matching and processing operations have clear data scope constraints, permission control conditions, and associated logical basis. Structured input not only improves the convenience and error tolerance of front-end user operations, but also provides a unified data model and trigger source for the back-end system's automated scheduling, permission filtering, and approval path configuration. This implements a design model where configuration is processing and input is rules, avoiding the redundant confirmation and communication required in traditional manual forms and email application processes, and improving the overall level of task automation.

[0085] S60, determining a data table associated field according to the data type field, matching the data value field of the date partition table with the data table associated field based on the target data table name field, and generating a matching result set;

[0086] In this embodiment, the data type field is used to clarify the semantic category on which the matching behavior is based, and its value determines in which type of basic data table the system should look for associated fields. Typical values ​​of the data type field include customer number, policy number, agent number, mobile phone number, ID number, etc., which are often preset as a finite enumeration set in system design to map different categories of data tables and their fields. After receiving the data type field, the system will parse out a corresponding data table associated field based on the internally configured field mapping table. This field is the target field for subsequent matching behaviors. For example, when the data type field is "policy number", the system automatically parses out the associated field as "policy_code"; when the data type field is "customer number", the associated field may be "customer_id".

[0087] The target data table name field indicates the name of the data source table corresponding to the current business processing target. This field can be used to filter data records from the date-partitioned table with consistent sources. In a date-partitioned table, the data value field typically carries the dataset submitted by the user for matching. Each record originates from raw input of a specific data type. After being synchronized and stored by the system, it is combined with the original synchronization date field and source table name field to form a complete record. The data value field itself does not carry semantics, but its semantic attributes can be identified through linkage with the data type field.

[0088] The matching operation uses data value fields as input and table-related fields as comparison targets. Before executing the match, the system requires two steps of data preparation: first, extracting the data value fields from the date-partitioned table, filtering based on the target table name field to ensure that only records in the corresponding target table are processed; second, extracting the table-related fields from the base data table to obtain their complete value list. The system then compares the data value fields against the table-related field value set one by one, typically using hashjoin, in-list, or semijoin methods. Successfully matched records are marked and output.

[0089] The output is presented as a matching result set. The result set structure should retain the target data table name field, the successfully matched data value field, the associated fields of the data tables involved in the comparison, the data type field, and other content. Data items that failed to match should be individually marked as unmatched and the reasons, such as "field is empty" or "not found in the system field," should be recorded for subsequent tracing. Once generated, the matching result set will become the foundational data for the security approval and permission filtering processes.

[0090] The system maintains the mapping between data type fields and associated fields in data tables based on configuration files or relational mapping tables. This mapping is resolved once data input is submitted. For example, if the data type field value is "ID Number," the system will query the table to confirm that its corresponding associated field in the data table is "person_id" and identify its underlying table as "customer_base." The system then uses the target data table's name field as a criterion to extract all records from the date-partitioned table where the source table's name field equals that value, and then extracts the data value fields within them. The data value fields represent the set to be matched, and the system must organize their contents in a list format, deduplicating them for subsequent comparison operations.

[0091] The system then connects to the target database and queries all unique values ​​for the associated fields in the underlying tables. A standard comparison set can be generated using statements such as "SELECT DISTINCT person_id FROM customer_base." The two sets are then compared using in-memory hash indexes or distributed data caching technology, forming two subsets: successful and unsuccessful matches. To ensure high concurrency and high throughput, the system can perform this comparison on platforms that support large-scale distributed processing, such as Spark, Presto, and ClickHouse. The matching process is carried out in batches, supporting multi-table parallel matching, automatic field type conversion, and null value cleaning strategies.

[0092] The resulting matching result set is stored in a temporary result table. The structure must retain fields such as the original data value, data type, matching results for table-related fields, and the target table name. This provides complete context for subsequent filtering, approval, and display. Unmatched records are stored in an exception record table or written to a log queue for consumption by the error analysis service.

[0093] Example: In financial business analysis, a data analyst needs to compare 50,000 customer IDs recently submitted from an external channel with the customer ID fields of valid insurance policies in the system to identify active customers in the system. Upon receiving the match request, the system extracts a list of customer ID values ​​from the partitioned table and matches them against the "insured_customer_id" field in the insurance policy base table. This generates a matching result set containing successfully matched customer numbers, corresponding policy numbers, and a list of unmatched customer numbers. This result set enters the approval process and is ultimately used for external compliance reporting or marketing plan screening.

[0094] In healthcare scenarios, disease control departments need to input a batch of ID numbers to determine whether these individuals have visited a fever clinic within the past two months. By setting the data type field to "ID number," the system automatically maps the "patient_id" field in the medical record table and performs high-frequency structured matching between the two. This matching result quickly identifies potential high-risk contact groups, providing automated support for prevention and control efforts and avoiding the delays and sensitive information exposure risks associated with traditional manual Excel checks.

[0095] By clarifying the mapping logic between data type fields and associated fields in the data table, a comparison relationship is established between the data to be matched and the underlying data. At the same time, the target data table name field is used to limit the scope of data to be processed, ensuring clear boundaries for matching behavior and reasonable permission control. The screening and structured organization of data value fields enables the batch matching process to be completed with an efficient data structure, while retaining unmatched records and detailed field context, enhancing the traceability and transparency of matching behavior. This process eliminates the inefficient process of manually querying and comparing data across multiple systems, making the matching logic automated, configurable, and supporting audit tracking, significantly improving the processing efficiency and security level of cross-table matching tasks.

[0096] S70, submitting a security sign-off including the user identification field and the matching result set through a report portal, and executing a security approval process according to the security sign-off;

[0097] In this embodiment, the construction of the security report revolves around the user identification field and the matching result set. The user identification field serves as an identity credential, enabling permission verification and accountability tracking during the approval process. The matching result set, as the core data carrier for the approval object, must ensure a reliable data source, complete structure, and clear field definitions. To prevent information leakage and content tampering, the system simultaneously constructs a report metadata summary during the security report generation process. This summary includes information such as the summary hash value, submission timestamp, and report number, which is used for subsequent integrity verification and approval validity verification.

[0098] Security sign-offs are submitted through the report portal. As the front-end entry point for user operations, the portal system packages the user identification fields with the matching result set to form a sign-off submission request, and records information such as the sign-off time, page source, and operator. During the submission process, the portal front-end performs basic validation, such as field content verification, format validation, and data packet size verification. After ensuring transmission validity, the sign-off request is sent to the security approval scheduling module.

[0099] The security approval process is driven by a collaborative effort between the approval queue and the policy module. The approval queue caches and queues pending sign-in requests. The policy module dynamically generates approval paths based on pre-set rules. These rules can be combined based on factors such as the user's organization, the sensitivity level of the matching result set, the data type field, or the scope of access permissions for data tables. The approval path is managed using a node graph structure. Each approval node is bound to a specific role, approval time limit, and processing priority. Approval tasks are triggered and transferred to the approver terminals at each level according to the path sequence.

[0100] Before each approval task is processed by the approver, the system generates a task data package. This package contains a user identification field, a summary of the matching result set, an approval rule code, and status information for upstream approval nodes. The task delivery mechanism supports multiple methods, such as email notifications and system message push notifications. The task status table records data such as the operation results, processing time, and approval opinions for each level of approval. When all approval nodes have completed the approval process and the approval status is consistently approved, the system generates an approval pass flag, which is used for subsequent data distribution and access rights triggering logic.

[0101] During actual deployment, the system frontend receives user input through a report portal interface built with Vue or React. It then calls an interface component to combine user identification fields with matching result sets into a structured report object. This structured object is serialized using JSON or ProtoBuf format, facilitating subsequent parsing and task management in the approval process. Before entering the approval module, the report data is first hashed and annotated with a submission timestamp using a signature encryption module to prevent tampering.

[0102] The security approval scheduling module builds an approval queue pool based on message queues (such as Kafka or RabbitMQ) and loads approval policy configurations through a rules engine. Approval policy configurations are defined using YAML or SQL structures, supporting dynamic loading and online adjustments. Field-level permission annotations in matching result sets affect the generation of approval paths. For example, if sensitive fields such as customer ID and ID number are included, a dedicated security approval node is automatically introduced.

[0103] After the approval path is generated by the approval policy module, it is stored and transferred through a graph database or nested linked list structure. Each node approval task is indexed by the approval task ID, and the task status is stored in a distributed KV storage service (such as Redis or Etcd) for rapid retrieval and writing of approval status in high-concurrency environments. When the task status changes to "passed" and is at the last node in the approval path, the approval scheduling module will write a status confirmation record and push the approval pass indicator to the data service module for subsequent processing.

[0104] Example: In the financial sector, a market risk manager initiates a match based on a customer's mobile phone number to confirm whether the specified number corresponds to a historical policyholder and obtain the relevant insurance type configuration. Once the match is complete, the system automatically generates a matching result set and encapsulates the user identification field and result set in the report portal to form a report request. Because the field associated with the mobile phone number is marked as highly sensitive, the approval strategy module automatically calls the security approval node and generates an approval path from "Department Head → Risk and Compliance → Security Approval". After approval is complete, the system generates an approval pass indicator and triggers the visual delivery of the result data.

[0105] In the healthcare sector, a hospital's compliance department wanted to verify fever clinic registration records within the past month based on ID numbers. The system automatically matched the data against the input, constructed a matching result set, and packaged a report. Because these records included highly sensitive fields like medical records and disease classifications, the system automatically introduced dedicated medical compliance and privacy officers for approval. The entire process was executed in real time through configurable policies, eliminating the need for manual submission of paper report approval forms and significantly improving the compliance and efficiency of data flow.

[0106] Through the full-process approval control of sensitive data matching results, from data submission to approval path generation, task assignment, approval flow to status confirmation after approval completion, the full-chain operation closed loop ensures the compliance of data use and the transparency of the approval process. With the precise mapping of user identification fields, it can be ensured that each request operation of the matching result can be audited and traced. The mechanism that automatically triggers the approval process after the matching result set is completed enables the system to avoid the tedious process of manually filling out approval forms and submitting approval requests offline, and realizes the automatic connection from matching behavior to approval action, improving the approval efficiency and security level. It effectively solves the problems of low approval efficiency, opaque execution path, and coarse permission control granularity in the traditional use of sensitive data.

[0107] S80, when the security approval process is completed, the matching result set is sent to the user terminal according to the binding relationship between the user identification field in the authority table and the name field in the target data table.

[0108] In this embodiment, the data transmission of the matching result set must be based on the clarity and verifiability of the permission binding relationship. The permission table records the binding mapping between the user identification field and the target data table name field. This binding relationship is generated after the user data table is created and the authorization configuration is completed. It is unique and immutable and is used to limit the scope of subjects with data access rights. After the approval process is completed, the system must re-verify whether the user identification field is still validly bound to the target data table name field involved in the matching result set to prevent the risk of data leakage after the authorization change.

[0109] As a structured data carrier that has been approved, the matching result set must be completely consistent with the previous matching logic, including technical features such as data value fields, data type fields, associated fields, and target data table name fields. Before sending the matching result set, the system will locate the permissions of the target data table name field. The positioning method is based on the permission table index structure for rapid retrieval to ensure that the data is only sent to user terminals with access qualifications. After the binding relationship verification is passed, the system calls the sending module to construct a transmission data packet with the matching result set in an encrypted form, and marks the matching task number, approval mark, and data usage flag during the transmission process for the user terminal to receive and track the results.

[0110] To further enhance controllability, the system records a complete log of each data delivery, including delivery time, data table name, user identification field, number of data items, and transmission status, and writes it to the audit system. This data can be used as evidence for tracing behavior in the event of subsequent access disputes, permission reviews, or security rollbacks. The receiving terminal decrypts the data packet and restores its structure to a user-readable result list. The current result data is archived according to the matching task number to support subsequent analysis or export.

[0111] In actual deployment, the matching result set structure is typically stored and transmitted in a columnar format, such as Parquet or ORC, to accommodate big data systems and distributed delivery services. During delivery, the system queries the permission table, indexed by the user ID field as the primary key, to confirm whether the user has a valid binding relationship with the target table name field in the matching result set. This binding relationship verification utilizes two logical checks: the existence of the creation authorization record and the validity of the record. If the binding relationship verification succeeds, the system generates a matching result data packet and encapsulates it into a ciphertext structure via the encryption service module. Data delivery utilizes either a push or pull strategy. The push strategy proactively sends data to the user's terminal's registered interface address via a gateway service. The pull strategy allows users to click a download link on the page after authentication, triggering limited-time access to the data interface. The interface service is deployed in the intranet access control zone and requires authentication using a combination of an access credential number and an approval ID. Data delivery is only possible after successful authentication. Abnormal transmission conditions, such as network interruptions, invalid destination addresses, and terminal authentication failures, are recorded as interruption events and trigger an alert module to be sent to data management personnel. For data that has not been successfully transmitted for a long time, an automatic retry queue can be set up and the number of retries can be limited to prevent resource usage and information leakage.

[0112] By introducing a binding relationship verification mechanism based on a permission table during data delivery, the system ensures that each matching result transmission has a legally authorized basis, avoiding the issue of incorrect data delivery due to binding failure or misoperation. Combining data encryption transmission, operation logging, and terminal authentication mechanisms, a closed-loop, controllable path from data generation to data distribution is formed, greatly enhancing the security and auditability of the data transmission process. Furthermore, by binding structured matching results to task numbers, the task management efficiency of user terminals can be improved, supporting a more sophisticated result archiving and permission management system.

[0113] The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. It discloses a data matching method based on security approval, including: creating a data table and configuring access rights, synchronizing the data in the table to an intermediate database after inserting the data into the collection table and adding field identifiers, synchronizing the intermediate database data to a distributed storage structure partitioned by date through a scheduled task; receiving the data table name, data type and identification information, determining the matching field based on the data type, matching the data value in the distributed storage with the target field and generating a result set; submitting the matching result as a security report and triggering the approval process, and sending the result data to the terminal according to the permission mapping after approval. The present invention realizes batch extraction, classification matching and permission compliance delivery of different types of data identification information by constructing a structured collection process, setting an automatic synchronization mechanism, performing data matching based on field type, and integrating approval control logic, thereby improving data processing efficiency and reducing the risk of sensitive data circulation.

[0114] In one embodiment, the above step S10 includes:

[0115] S101, receiving a data structure definition including a data sequence number field, a data type field, a data value field, and a user identification field;

[0116] S102, creating a user data table based on the data structure definition, and writing the data sequence number field, data type field, data value field, and user identification field into the user data table structure information according to a preset field sequence;

[0117] S103, generating a user data table name field corresponding to the user data table according to the user identification field and a preset naming strategy;

[0118] S104, verifying whether the data sequence number field in the user data table is a numeric field, whether the data type field is within a fixed value range, whether the data value field is a character field, and whether the user identification field complies with a preset format policy;

[0119] S105, when the field type or field order verification of the data sequence number field, data type field, data value field or user identification field fails, the user data table creation operation is terminated and an error log containing an error type identifier is generated;

[0120] S106, when the user data table is created, the binding relationship between the user data table name field and the user identification field is written into the permission table;

[0121] S107, granting the collection user account the permission to read the user data table and the permission to write the collection data table, generating a unique access credential number and establishing an association with the user identification field;

[0122] S108: Return the unique access credential number to the user terminal, and configure an access control policy to restrict access to the user data table to only requests carrying a valid access credential number.

[0123] In this embodiment, when the system receives the table structure definition information, it must first complete the format parsing and field classification of the input structure. Data structure definition refers to a structured object that contains field name, field type, and field order, which is used to clarify the precise configuration of field arrangement when building a table in the future. Among them, the data sequence number field is usually numeric data, which is used to identify the order or uniqueness of data records; the data type field defines the data type to which the data value field belongs, such as "customer number", "policy number", and "agent number". The value of this field is limited to a fixed value set preset by the system to ensure the standardization of subsequent matching logic; the data value field is the business identification information to be analyzed, and its type is set to character type to be compatible with different encoding systems; the user identification field is an identity information field used to distinguish table ownership, which is usually bound to the enterprise UM number or authentication account.

[0124] After parsing the structure definition, the system completes field mapping according to the preset field order and builds it into the database table creation statement. The preset field order strategy ensures that all user data tables have a completely consistent structure, which facilitates the programmatic invocation of subsequent matching processes and the uniform format of result sets.

[0125] The system then generates the user data table name field based on the value of the user ID field and a naming template. For example, the system can automatically name the table using the combination "prefix_user ID_timestamp", ensuring unique table names within the physical database and enhancing manageability.

[0126] After the table is created, the system initiates a field type and value range verification process, verifying the data structure validity of each field one by one. This verification includes: confirming that the data sequence field is an integer or long integer; verifying that the data type field is within the allowed enumeration value range; ensuring that the data value field can accept variable-length characters; and verifying that the user identification field format meets the company's unified internal standards (such as email or UM number format). If any field does not meet these requirements, the system will reject the table creation request and generate an error log. The log record includes information such as the name of the failed field, the reason for failure, and a timestamp, allowing the user to correct the failure and try again.

[0127] After verification, the system writes the table name and user ID field of the user data table into the permission table, forming a binding relationship. This binding record is the core judgment basis for subsequent data identification, permission verification, and data distribution.

[0128] Finally, the system grants the account where the collection program is located read permissions for the data table and write permissions for the collection data table. To support access tracing and authorization tracking, the system also generates a unique access credential number and binds this number to the user identification field. The access credential number is encapsulated in encrypted form when it is sent to the user terminal. The user needs to use this credential to initiate data processing requests in subsequent operations. The system configures access control policies based on the credential, clearly allowing only requests with valid credentials to access the corresponding user data table, thereby establishing an access boundary based on the credential number.

[0129] During implementation, data structure definitions can be automatically parsed by users uploading JSON or Excel templates. The parser uses a field mapping mechanism to compare uploaded field names with the system's internal field templates to ensure that the field content and order meet the construction requirements. The data management platform automatically generates and executes DDL statements for table creation, using automatic table name concatenation rules to generate unique table identifiers.

[0130] For field validation, the system uses a metadata-based field type verification module, which runs during the pre-check phase before table creation. This module uses the system field dictionary to perform type identification and regular expression validation on input field data. If type conflicts, empty fields, or non-compliant values ​​are detected, an exception is thrown and table creation is terminated.

[0131] Permission tables can be implemented using a high-performance KV structure or a relational table index structure to ensure fast matching between user identification fields and table names. Permission granting is automatically executed through database GRANT statements, and the Permission Policy Center generates access credential numbers. These numbers are generated in encrypted UUID format and feature timestamps, single-use binding, and uniqueness.

[0132] The access control policy is deployed at the service gateway layer. The gateway module verifies whether the credential number contained in the request header is valid and whether it matches the current user identification field. If the verification fails, access is denied to ensure that the data table access boundaries are strictly controlled.

[0133] This embodiment constructs a unified format data structure definition process, so that the system can standardize the table structure of user data tables and improve the consistency of data processing; strict verification of field types and sequences avoids data interruption problems caused by structural errors or non-compliant data; naming rules and permission binding logic enhance the traceability of table management and data isolation effects; the generation and binding of unique access credentials enables subsequent data access to have clear permission identification and security control boundaries, thereby providing a basic and reliable starting point for the entire matching data processing system, avoiding manual intervention, misuse and illegal access.

[0134] In one embodiment, the above step S20 includes:

[0135] S201, querying the user data table name field bound to the current user identification field in the permission table, and generating a list of user data tables authorized by the current user based on the user data table name field;

[0136] S202, verifying whether each user data table name field in the user data table list exists in the database table directory;

[0137] S203, when the user data table name field exists in the database table directory, reading data records from the user data table according to a preset paging size;

[0138] S204, performing a field-level format conversion operation on each read data record;

[0139] S205, inserting the format-converted data records into the collection data table in batches, and generating a data insertion transaction log when each batch of data records is inserted;

[0140] S206: When a data record insertion failure is detected, the unfinished batch of data record insertion operations is rolled back according to the data insertion transaction log and an alarm notification is triggered.

[0141] In this embodiment, identifying the data tables to which the current user has access permissions is fundamental to ensuring compliance in the data collection process. The permissions table is a structure containing bindings between user ID fields and table name fields. When querying, the system first filters all binding records based on the current user ID field, forming a set of authorized user data table name fields. This set represents the range of data sources that the current user is permitted to access and has collection permissions for.

[0142] Each data table name field represents a logically independent container of structured data. Before executing the collection process, the system verifies that these tables actually exist in the target database directory. This can be accomplished by querying the database system tables, data dictionary, or calling the metadata service interface. Subsequent collection and reading operations will only begin if the table name exists in a valid directory of the current data source. This prevents data anomalies or task interruptions caused by inconsistencies between logical layer records and physical table states.

[0143] To ensure processing performance and data security, the system uses a paging mechanism when reading user data tables. The paging size is a system parameter setting, typically in batches of 1,000 to 10,000 records, depending on the actual database load and network transmission limitations. Paging not only reduces single-time memory consumption but also supports task recovery and state management, ensuring system controllability when processing large amounts of data.

[0144] After reading each data record, the system immediately performs field-level format conversion. Field-level format conversion refers to the standardization of specific field values ​​to meet the system's unified structure requirements. Specifically, the data sequence number field is converted to a unified numeric format to ensure that subsequent sorting, positioning, and deduplication operations can be performed based on consistent data types. Full-width characters in the data value field are individually identified and replaced with their corresponding half-width characters. This operation is often performed using a character encoding mapping table to ensure data consistency and the accuracy of subsequent comparison calculations.

[0145] After conversion, the system writes data records in batches to the collection data table. The collection data table serves as an intermediate container for all standardized records from user tables. Data insertion is based on a transaction control model. After each batch of data is inserted, a data insertion transaction log is generated. The log contains key information such as the number of inserted data entries, field mapping information, timestamp, and source table name. This logging mechanism provides the basis for subsequent retry attempts, rollbacks, and audits.

[0146] If a batch of data records fails during insertion, such as due to a primary key conflict, field type anomaly, or database write anomaly, the system automatically performs a rollback based on the generated transaction log, deleting the inserted but incomplete records in the current batch and triggering an alarm. This alarm mechanism then links with the system notification center to send an error message to platform administrators and data processing personnel. The message includes the batch number, cause of the anomaly, and source table information, enabling subsequent investigation and reprocessing.

[0147] In actual deployment, permission tables can be centrally maintained using a relational database or distributed metadata service. Each time a data collection task is initiated, the system calls the permission service module to retrieve the authorization table list corresponding to the current user's identification field. The system supports an asynchronous validation mechanism to concurrently verify the existence of tables in the database metadata directory, improving efficiency. Paged read operations are scheduled by the batch processing engine. During each page, the system stores the read rows in a memory cache structure. During the field-level conversion phase, the format conversion module is called to perform data sequence number normalization (such as conversion to int64) and character replacement. The character replacement module uses mapping tables or regular expressions for efficient implementation. Data collection table writes are executed via batch insert SQL, wrapped by the transaction control module. After each batch insert, the log generator is called to write to the log archive table. The rollback mechanism uses the primary key or hash fingerprint of the written record to check the log, clean up incomplete batches using the DELETE operation, and push notifications to the operation and maintenance platform and management platform in the exception path, or trigger the webhook process to support further automated processing.

[0148] This embodiment ensures the legitimacy of data sources through permission binding verification, and the paging processing and field conversion strategies enhance the system's processing capabilities and data consistency; the transaction log and rollback mechanism improves the system's abnormal recovery capabilities and data integrity; the alarm mechanism enhances fault observability and data processing security, thereby realizing a highly available, traceable, and recoverable data collection mechanism for structured batch data, providing a high-quality data foundation for subsequent matching processing.

[0149] In one embodiment, the above step S40 includes:

[0150] S401, extracting the user data table name field, data value field, and synchronization date field from the intermediate database, and serializing them into columnar storage structure data;

[0151] S402, generating a partition identifier according to the date value of the synchronization date field, wherein the partition identifier is hierarchically organized into a year directory, a month directory, and a day directory;

[0152] S403: Write the serialized columnar storage structure data into the date partition table of the distributed storage according to the preset batch size. Generate an independent data file for each batch of columnar storage structure data and store it in the directory of the corresponding partition identifier.

[0153] S404, performing a data integrity check on the written data file to verify whether the total number of records in the user data table name field, data value field, and synchronization date field in the date partition table is consistent with the total number of synchronized records in the intermediate database; if the verification passes, marking the data synchronization status as completed;

[0154] S405: According to a preset data retention period policy, the year directory, month directory, date directory and data files stored in the corresponding directories in the date partition table that exceed the retention period are deleted.

[0155] In this embodiment, the intermediate database serves as a transition layer for data written from the collected data table, providing a structured and preliminarily cleansed set of records for subsequent data processing. The intermediate database must contain at least the following fields: the user data table name field, the data value field, and the synchronization date field. The user data table name field identifies the data source, the data value field carries core identification information used for subsequent matching, and the synchronization date field identifies the most recent processing time for the data, forming the basis for performing time-dimension partitioning.

[0156] Before synchronizing data to distributed storage, the aforementioned fields in the intermediate database must be extracted and converted to a columnar storage structure. Columnar storage is a data storage format optimized for analytical computing. Compared to row-based storage, it more efficiently supports field filtering, compression encoding, and batch loading. This structure typically uses standard formats such as Parquet and ORC. The serialization process includes operations such as field type mapping, encoding compression, and field reordering to ensure maximum subsequent read performance.

[0157] The value of the synchronization date field is used as the key anchor point for generating data partition identifiers. To support hierarchical data access and storage organization, partition identifiers are structured as a three-level directory structure: year, month, and day, organized in a progressive manner, forming a path such as / 2025 / 04 / 06 / . This structure not only improves query efficiency but also facilitates periodic cleanup operations.

[0158] Synchronization operations utilize a batch control strategy, with the maximum number of rows processed per batch set by system configuration parameters. During each batch synchronization, serialized data is written as files to a distributed storage system (such as HDFS, object storage, or a lake-warehouse integrated platform) and stored in the corresponding partition directory. Each data file has a unique naming convention (such as data batch number and write timestamp), supporting task idempotence control and log traceability.

[0159] To ensure data integrity during synchronization, the system performs a consistency check after each batch of data files is written. This check compares the number of records written to the distributed storage directory with the number of records from the same batch in the intermediate database. It can further verify that field value distribution meets specifications or verify file content integrity by comparing data hash values. If the check passes, the data status is recorded as "Synchronization Completed" and written to the data synchronization task log for subsequent use.

[0160] Considering data storage costs and data compliance requirements, the system supports data retention cycle policies for partitioned directories. Based on a preset period (e.g., 30 or 90 days), the time difference between the current time and the partition corresponding to the synchronization date field is determined. Directories (year, month, day) that fall outside the retention time range and their subordinate data files are automatically identified and cleared. This process can be implemented through scheduled cleanup tasks, policy drivers, or event-triggered mechanisms, ensuring that data storage pressure is controlled during long-term system operation.

[0161] In one specific implementation, the system reads three columns of field data from the intermediate database through SQL or a data interface, and passes the results to the serialization engine for processing. The engine converts it into Parquet format based on the field definition, and applies a column compression algorithm (such as Snappy, ZSTD), ultimately forming data blocks organized by columns and writing them to the local buffer path. The write scheduler allocates the batch of data to a path such as / 2025 / 04 / 06 / based on the value of the synchronization date field, and completes the final storage through a distributed writing tool (such as Spark, Flink Sink, DataX). The batch number or task ID is appended to each data file name to ensure that there is no file conflict. After the write is completed, the metadata service is called to count the number of rows in the current partition data file, and then compares it with the number of intermediate database records recorded when the task is submitted. If inconsistency is found or verification fails, the current task mark is rolled back and the compensation mechanism process is entered to prevent erroneous data from being exposed to the downstream. The data retention policy can be set through configuration parameters. For example, if the retention is set to 30 days, the daily midnight scheduling task will scan all partition directories under the current storage path, compare them with the system time, and delete all directories that are 30 days older than the current date to ensure that the data life cycle complies with the governance policy.

[0162] This embodiment supports the unification of field structures, standardization of access paths, and dynamic management and control of storage resources through a stable transition from a structured intermediate database to high-performance distributed storage. During the structural conversion process, a columnar structure is utilized to improve storage compression efficiency and analytical processing performance; multi-layer partitioning is used to significantly improve data query and location efficiency; batch control and transaction verification are combined during the write phase to enhance data consistency assurance capabilities; and lifecycle policies are used to control the release of redundant data, ensuring the economic efficiency and stability of the platform's long-term operation.

[0163] In one embodiment, the above step S60 includes:

[0164] S601, determining the name of the base table to be associated and the associated fields in the corresponding base table according to the value of the data type field;

[0165] S602, based on the target data table name field, filter out data records whose source table name field is equal to the target data table name field from the date partition table;

[0166] S603, extracting the data value field corresponding to the data type field from the filtered data records to generate a data set to be matched;

[0167] S604, based on the name of the basic table and the associated fields, query the target database for a complete value list of the associated fields in the basic table;

[0168] S605, matching the data value field in the to-be-matched data set with the complete value list of the associated field, and filtering out the target data value field that exists in the complete value list;

[0169] S606: Generate a matching result set based on the target data table name field, target data value field, and associated fields.

[0170] In this embodiment, the data type field serves as the core input driving the field mapping logic, reflecting the identification type that the business side is actually concerned about, such as "policy number", "customer number" or "agent number", and its enumeration value set is usually fixed and preset in the system. Based on the value of this field, the target data source that the current matching task should rely on can be determined, that is, the basic base table and the unique fields within it. The basic base table is a predefined structured data set with clear field mapping relationships for fields of different data types. For example, "policy number" corresponds to the policy number field in the policy base table, and "customer number" corresponds to the customer number field in the customer base table. This mapping relationship is defined by a configuration template or a dictionary table, and has good maintainability and scalability.

[0171] The target data table name field is a front-end submission parameter that identifies the data source table that matches the current user's intent. In date-partitioned tables in distributed storage, each record has a source table name field, which stores the identifier of the original user data table to which the record belongs. The system filters the target data table name field against the source table name field for equality, accurately extracting the original data records corresponding to the matching task and establishing the data processing context.

[0172] From the filtered records, we further extract the column of field values ​​corresponding to the current data type field to form a structurally consistent set of data values ​​for subsequent batch matching operations. To ensure matching efficiency and maintainability, the data value field maintains a consistent structure with the associated field type, and the matching method is exact matching, without introducing fuzzy logic.

[0173] The complete value list for the associated fields refers to the on-demand extraction of all primary key or unique index fields from the underlying table that are not null and meet the filtering criteria from the target database. This list can be obtained dynamically by constructing a query statement or calling an API. To improve performance, this process can use field deduplication, caching mechanisms, or field statistical indexes to optimize query time.

[0174] The matching process is completed using a set intersection method, comparing the data value fields in the matching dataset with the associated field value set retrieved from the underlying table, screening out complete hit records as valid matches. At the same time, to enhance data transparency and traceability, unmatched field values ​​are recorded and written, along with the corresponding data type field information, into a separate match failure record table or log structure to support subsequent user verification and policy adjustments.

[0175] Ultimately, the matching result set must retain the successfully matched target data value field, its corresponding target data table name field, and the identification information of the associated field, so that the matching source and basis can be traced when subsequently generating approval requests or outputting reports.

[0176] This embodiment dynamically binds the target data table name field passed in by the front-end user to the source table fields in the partitioned data, and reversely derives the underlying table and associated fields based on the data type field. This allows the system to automate and accurately compare data without requiring user-configured field mapping rules. Furthermore, matching results and failed records are stored separately, facilitating auditing and optimization while preventing subsequent data streams from being contaminated by erroneous data, thereby improving the overall data security and robustness of the system.

[0177] In one embodiment, after the above step S60, the method further includes:

[0178] S607: Receive the organizational unit code of the user terminal on the report page, and generate an authorized organizational unit list based on the organizational unit code and a preset organizational structure tree relationship;

[0179] S608, extracting the data records to be screened that contain the organizational unit field associated with the authority from the matching result set;

[0180] S609 , based on the authorized organizational unit list, filter out target data records whose values ​​of the organizational unit fields belong to the authorized organizational unit list from the data records to be filtered, and generate a final matching result set based on the target data records.

[0181] In this embodiment, the organizational unit code is used to characterize the identity and authority of the data user in the organizational structure. It is one of the key fields in the enterprise data authority control system. It is usually represented by a structured code (such as OU001, OU001A) and has uniqueness and hierarchy. This code is an accompanying attribute of the user request and is bound to the user login information or page identity context. The organizational structure tree relationship defines the hierarchical relationship between organizational units at all levels. It is usually represented in the database as an adjacency list, path enumeration or nested set model to support authority inheritance and cross-level access control.

[0182] The authorized organizational unit list is a collection of permission scopes calculated based on the organizational unit code of the current user terminal and the organizational structure tree. This list includes not only the current code itself but also the codes of its subordinate departments. This list can be generated through recursive queries or based on the organizational structure cache. Sometimes, the scope of authorization can be further restricted through policy configuration.

[0183] The initial matching result set accurately matches the data value fields with the associated fields in the data table. Within this result set, the system ensures that each data record includes an organizational unit field, indicating the actual organizational affiliation of the data. Common organizational unit fields include "Sales Organization Code" and "Business Department Number." While their names vary depending on the business scenario, their functionality remains the same: they identify the organizational dimension of the data.

[0184] The data records to be filtered are the subset of all records in the matching result set that contain the Organization Unit field. To ensure the integrity of permission filtering, the system performs a non-null check on this field and excludes records that are missing the Organization Unit field from the set to be filtered.

[0185] Target data records are those whose Organization Unit field values ​​are in the Authorized Organization Unit List. This filtering operation is typically implemented using a set filter. If multiple code values ​​exist in the Authorized Organization Unit List, the system compares each one individually using methods such as "IN" matching or hash lookup to improve processing performance.

[0186] The final matching result set is the screening result after introducing organizational unit permission filtering on the basis of the original matching result set. Only the target data records are retained for subsequent display, approval or distribution, ensuring that the data can only be accessed and processed by authorized organizational users.

[0187] In one specific embodiment, after receiving the matched data, the system extracts the organizational unit code of the current user, such as "OU001", from the front-end user context. According to the organizational tree structure, the user is identified as a provincial agency under a head office, which contains multiple business outlets (such as "OU001-A", "OU001-B"). The system generates an authorized organizational unit list [OU001, OU001-A, OU001-B] through the path enumeration table as the scope of authority. Subsequently, all records containing the "affiliated organization number" field are filtered out from the matching result set to form a data set to be filtered. During the screening process, the system verifies that the field is not empty and that the field type is a standard structured coding format. For records with missing fields or incorrect formats, the system can classify them as records that "do not have authority attributes", and subsequent modules will decide whether to block or mark them. The system completes the authority filtering operation based on the logic of "affiliated organization number IN authorized organizational unit list". After screening, only the data belonging to "OU001" and its subordinate institutions are retained in the matching result set to generate the final matching result set for subsequent report submission, data export or system display.

[0188] By introducing the organizational unit dimension into the data screening logic, this embodiment enables data access control accurate to the organizational level, effectively preventing data leakage or unauthorized use. Compared to the traditional full result set output method, the introduction of permission screening ensures that the same matching logic has differentiated data visibility on different user sides, strengthening the system's multi-tenant isolation and data compliance capabilities. This mechanism is particularly suitable for business scenarios such as regulatory data distribution and sensitive data query, and has strong practical feasibility and universal scalability.

[0189] In one embodiment, the above step S70 includes:

[0190] S701, generating a security report request including the user identification field, the matching result set and the report submission timestamp in the report portal;

[0191] S702: Submit the security sign-off request to a preset security approval queue, wherein the security approval queue is associated with multiple levels of approval nodes and approval role permissions;

[0192] S703: The approval policy module determines the approval path for the security sign-off request based on the permission attribute of the user identification field, the preset approval policy in the approval policy library, and the metadata of the security sign-off request, and triggers the corresponding approval node to generate an approval task.

[0193] S704, recording the operation log of each approval node in the approval process, and generating an approval pass mark when all approval nodes complete the corresponding approval tasks and the approval status is passed.

[0194] In this embodiment, a security sign-off request refers to a packaged structure generated on the report portal's frontend before data is released, enabling compliance review and traceability control throughout the data access process. This request structure includes a user ID field, used to identify the user initiating the sign-off operation; a matching result set, which carries the specific data content generated through previous data matching and permission screening steps; and a sign-off submission timestamp, which marks the initiation time of the data sign-off request and serves as a basis for time-sensitive approval control and tracking comparison.

[0195] The security approval queue is a structured task management mechanism used to cache and manage pending approval requests. It's typically hosted by a messaging middleware or persistent task scheduling system. This queue schedules approvals on a first-in-first-out or priority-based basis. Each request in the queue is tied to an approval path structure, ensuring a controllable approval order and consistent triggering logic across approval nodes.

[0196] The approval policy module determines the approval process path based on the contextual attributes of the current sign-in request. Its core logic includes parsing the user identification field to determine the user's permission level, job role, or organizational affiliation. It then combines the rules set in the approval policy library to determine whether second- or third-level approval is required based on data sensitivity, matching data size, or target usage. It also determines the role requirements and approval action types for each level of approval.

[0197] The approval path is a chain of approval nodes from the generation of a sign-off request to the final data approval. Each node is uniquely identified by an approval node identifier. The approval node identifier is used to indicate the node in the current process where the approval action is performed. For example, the first-level approval node may be the head of the data ownership department, while the second-level approval node may be composed of representatives from the compliance department.

[0198] An approval task is an approval operation unit generated by the approval policy module and dispatched to the approval queue for execution. It contains a user identification field for tracking the responsible party, an approval node identification for indicating the current task location, an approval path rule code for restoring the logical structure of the approval link, and a security report request summary hash value for identifying the content of the approval object to prevent data from being tampered with midway and support audit comparison.

[0199] Approval task status records include status indicators such as pending approval, approved, rejected, and withdrawn. The system updates the status field after each node's approval operation is completed and records the operation log. Log fields typically include the approver's ID, approval opinion, approval timestamp, and operating device information. The approved indicator is used to mark the entire approval path as fully executed and passed. This indicator is triggered after the approval process is successfully closed and serves as a prerequisite for the final release or decryption of data.

[0200] The approval notification message is pushed by the system to the corresponding approver's terminal based on the approval node identifier of the approval task. The notification content may include the approval task summary, task number, deadline, etc., to remind the user to complete the approval operation in a timely manner.

[0201] Example: In the fintech sector, the risk control department is conducting a special investigation into agent behavior risks. The business team has a collection of agent numbers and urgently needs to correlate and query the number of insurance policies, policy amounts, financial blacklist status of insured customers, and credit profile of each agent. This data is stored in separate system tables and contains sensitive customer fields, making manual query inefficient and subject to access control restrictions. The operator first submits a data structure definition through the data platform front-end, including a data sequence number field, a data type field (such as "agent number"), a data value field (a list of specific agent numbers), and a user identifier field. The platform automatically creates a user data table with a consistent data structure, configures permissions, and associates the user with the table name. The system recognizes that the user has collection permissions. The task scheduling module periodically extracts data from the user table, cleans and transforms it, and writes it to the collection data table. Once the data is imported into the intermediate database, the system automatically annotates each record with the source user table name and the data synchronization time. The data is then synchronized to a date-partitioned table in the distributed storage, stored by day and automatically partitioned for easy querying and management. A business operator submits a matching request through the report page, specifying the target data table name field as the current table and the data type field as "agent number." Based on this field, the system identifies the core associated field in the agent system as "agent_code." It extracts the data value fields corresponding to the agent number from the date-partitioned table and performs a batch match against the agent master data table to generate a matching result set. The system detects that the data contains sensitive customer information (such as customer ID number and financial tag score). Based on organizational permissions, the user is restricted to viewing results within their business group. The system further generates a list of authorized organizational units based on their organizational unit code, filtering the matching result set to only rows belonging to that organization, generating the final matching result set. The platform then automatically constructs a security report request, packaging the matching result set, user identification fields, and submission time into a report structure, and submits it to the approval queue. Based on user role and data sensitivity, the system dynamically generates a three-level approval path: data ownership department head, compliance officer, and information security officer. Approval tasks are pushed to the approval terminal layer by layer, with approval comments and task status recorded throughout the process, ultimately resulting in an approved status. After the approval is completed, the platform system confirms the legitimacy of the user to access the current target data table through the binding relationship recorded in the permission table, and securely sends the final matching result set to the user interface for export, analysis or use in subsequent risk model input.

[0202] In the healthcare sector, the Medical Statistics Department is conducting a nationwide project to assess the under-registration rate of chronic disease follow-up. The department has obtained a batch of medical insurance or ID card numbers and needs to identify whether these patients have previously visited the institution and analyze their previous diagnoses, hospitalizations, and medical insurance reimbursement status. Considering the sensitivity of medical data and privacy protection constraints, the project leader first created a structured data input template on the platform. The data type field is "Medical Insurance Number" and the data value field is the specific patient number, along with the department identifier. Based on the structure definition, the system automatically creates a patient query table and verifies the validity of the field format, data range, and field order. Once created, the system verifies that the user has access to the current medical analysis module through the permission table and initiates a scheduled synchronization process. The medical insurance number data in the patient table is written to the collection data table, cleaned, and transferred to the intermediate database, along with the data source table name and synchronization timestamp. Daily data is automatically archived in a distributed storage structure, logically partitioned by year / month / day. The project leader then enters the target data table name field and data type field in the analysis platform interface. The system determines that the primary key field in the diagnosis and treatment information system to be associated is "insurance_id" based on the medical insurance number field, extracts the data value corresponding to this field in the partition table, and queries the associated data from the diagnosis and treatment system, including diagnostic records, drug use, examination frequency, etc. After the matching results are formed, the platform automatically filters according to its department code, retains only the data within the authorized scope, and forms it into the final result set. Given that the data content involves multiple patients' personal medical records and medical insurance records, the system automatically submits the report to the hospital's ethics review committee, safety management department, and information center director for approval. The approval task is distributed to the corresponding approver end, and the process is marked as approved after completion. Finally, the platform confirms that the project leader can access the target data table through the configured permission table, and automatically pushes the data to its analysis view or table export module for the writing of the omission rate analysis report and the submission of policy recommendations.

[0203] This embodiment builds a secure sign-off request structure centered around user identification fields, matching result sets, and sign-off timestamps. This system achieves full-link data approval traceability and approval chain integrity. By introducing a multi-level approval path mechanism and approval policy module, the system dynamically matches the sensitivity level of data access requests with the strength of the approval path, significantly reducing the risk of unauthorized access and compliance risks. By identifying the content of matching results with hash values, approval tasks ensure consistency and content integrity verification, forming an auditable closed-loop operation. This process significantly improves the automation and security level of the sensitive data access approval process while alleviating the complexity of maintaining manual approval paths.

[0204] In one embodiment, a data matching device based on security approval is provided, and the data matching device based on security approval corresponds one-to-one with the data matching method based on security approval in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a data matching device based on security approval according to the present invention. It includes a user data table configuration module 10, a data acquisition and processing module 20, an intermediate data synchronization module 30, a distributed storage management module 40, a report input and reception module 50, a data matching execution module 60, a security report approval module 70, and a matching result sending module 80. Each functional module is described in detail below:

[0205] A user data table configuration module 10 is used to create a user data table and set access authorization for the user data table;

[0206] The data collection and processing module 20 is used to identify the authorized user data table through the permission table and insert the data in the user data table into the collection data table;

[0207] An intermediate data synchronization module 30 is used to set a scheduled task to synchronize the data in the collected data table to an intermediate database, and to add a source table name field and a synchronization date field to the intermediate database;

[0208] A distributed storage management module 40 is configured to synchronize the data in the intermediate database to a date partition table of distributed storage, and to establish partition storage for the date partition table based on the synchronization date field;

[0209] The report input receiving module 50 is used to receive input content of the user identification field, the target data table name field and the data type field on the report page;

[0210] A data matching execution module 60 is configured to determine a data table associated field according to the data type field, match the data value field of the date partition table with the data table associated field based on the target data table name field, and generate a matching result set;

[0211] A security sign-on approval module 70 is configured to submit a security sign-on containing the user identification field and the matching result set through a report portal, and execute a security approval process based on the security sign-on;

[0212] The matching result sending module 80 is used to send the matching result set to the user terminal according to the binding relationship between the user identification field in the permission table and the name field in the target data table when the security approval process is completed.

[0213] In one embodiment, the user data table configuration module 10 is specifically configured to:

[0214] Receive a data structure definition including a data sequence number field, a data type field, a data value field, and a user identification field;

[0215] Create a user data table based on the data structure definition, and write the data sequence number field, data type field, data value field and user identification field into the user data table structure information according to a preset field order;

[0216] Generate a user data table name field corresponding to the user data table according to the user identification field and a preset naming strategy;

[0217] Verify whether the data sequence number field in the user data table is a numeric field, whether the data type field is within a fixed value range, whether the data value field is a character field, and whether the user identification field complies with a preset format strategy;

[0218] When the field type or field order verification of the data sequence number field, data type field, data value field or user identification field fails, the user data table creation operation is terminated and an error log containing the error type identifier is generated;

[0219] When the user data table is created, the binding relationship between the user data table name field and the user identification field is written into the permission table;

[0220] Granting the user account of the collection user the permission to read the user data table and the permission to write the collection data table, generating a unique access credential number and establishing an association with the user identification field;

[0221] The unique access credential number is returned to the user terminal, and an access control policy is configured to restrict access to the user data table to only requests carrying a valid access credential number.

[0222] In one embodiment, the data acquisition and processing module 20 is specifically configured to:

[0223] Query the user data table name field bound to the current user ID field in the permission table, and generate a list of user data tables that the current user has authorized based on the user data table name field;

[0224] Verify whether each user data table name field in the user data table list exists in the database table directory;

[0225] When the user data table name field exists in the database table directory, data records are read from the user data table according to the preset paging size;

[0226] Perform field-level format conversion on each data record read;

[0227] Insert the format-converted data records into the collection data table in batches, and generate a data insertion transaction log when each batch of data records is inserted;

[0228] When a data record insertion failure is detected, the unfinished batch of data record insertion operations is rolled back according to the data insertion transaction log and an alarm notification is triggered.

[0229] In one embodiment, the distributed storage management module 40 is specifically configured to:

[0230] Extract the user data table name field, data value field and synchronization date field from the intermediate database, and serialize them into column storage structure data;

[0231] Generate a partition identifier according to the date value of the synchronization date field, wherein the partition identifier is organized into a year directory, a month directory, and a day directory in a hierarchical structure;

[0232] Write the serialized columnar storage structure data into the date partition table of distributed storage according to the preset batch size. Generate an independent data file for each batch of columnar storage structure data and store it in the directory identified by the corresponding partition.

[0233] Performing a data integrity check on the written data file to verify whether the total number of records in the user data table name field, data value field, and synchronization date field in the date partition table is consistent with the total number of synchronized records in the intermediate database; if the verification passes, marking the data synchronization status as completed;

[0234] According to the preset data retention period policy, the year directory, month directory, date directory and data files stored in the corresponding directories in the date partition table that exceed the retention period are deleted.

[0235] In one embodiment, the data matching execution module 60 is specifically configured to:

[0236] Determine the name of the base table to be associated and the associated field in the corresponding base table according to the value of the data type field;

[0237] According to the target data table name field, filter out data records whose source table name field is equal to the target data table name field from the date partition table;

[0238] Extracting the data value field corresponding to the data type field from the filtered data records to generate a data set to be matched;

[0239] Based on the name of the basic table and the associated fields, query the target database for a complete list of values ​​for the associated fields in the basic table;

[0240] Matching the data value field in the to-be-matched data set with the complete value list of the associated field, and filtering out the target data value field that exists in the complete value list;

[0241] A matching result set is generated based on the target data table name field, target data value field and associated fields.

[0242] In one embodiment, the data matching execution module 60 is specifically configured to:

[0243] Receive the organizational unit code of the user terminal on the report page, and generate an authorized organizational unit list based on the organizational unit code and the preset organizational structure tree relationship;

[0244] Extracting the data record to be screened that contains the organizational unit field associated with the authority from the matching result set;

[0245] Based on the authorized organizational unit list, target data records whose values ​​of the organizational unit fields belong to the authorized organizational unit list are screened out from the data records to be screened, and a final matching result set is generated based on the target data records.

[0246] In one embodiment, the security sign-off approval module 70 is specifically configured to:

[0247] Generate a security report request in the report portal that includes the user identification field, the matching result set, and the report submission timestamp;

[0248] Submit the security sign-off request to a preset security approval queue, which is associated with multiple levels of approval nodes and approval role permissions;

[0249] The approval policy module determines the approval path of the security sign-on request based on the permission attribute of the user identification field, the preset approval policy in the approval policy library, and the metadata of the security sign-on request, and triggers the corresponding approval node to generate an approval task;

[0250] The operation log of each approval node is recorded in the approval process. When all approval nodes complete the corresponding approval tasks and the approval status is passed, an approval pass mark is generated.

[0251] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a data matching method based on security approval on the server side.

[0252] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a data matching method based on security approval.

[0253] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0254] Create a user data table and set access authorization for the user data table;

[0255] Identify the authorized user data table through the permission table, and insert the data in the user data table into the collection data table;

[0256] Setting a scheduled task to synchronize the data in the collected data table to an intermediate database, and adding a source table name field and a synchronization date field to the intermediate database;

[0257] Synchronize the data of the intermediate database to the date partition table of distributed storage, and establish partition storage for the date partition table based on the synchronization date field;

[0258] Receive input content of the user identification field, the target data table name field, and the data type field on the report page;

[0259] Determine the data table associated field according to the data type field, match the data value field of the date partition table with the data table associated field based on the target data table name field, and generate a matching result set;

[0260] Submitting a security sign-off including the user identification field and the matching result set through a report portal, and executing a security approval process based on the security sign-off;

[0261] When the security approval process is completed, the matching result set is sent to the user terminal according to the binding relationship between the user identification field in the authority table and the name field of the target data table.

[0262] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0263] Create a user data table and set access authorization for the user data table;

[0264] Identify the authorized user data table through the permission table, and insert the data in the user data table into the collection data table;

[0265] Setting a scheduled task to synchronize the data in the collected data table to an intermediate database, and adding a source table name field and a synchronization date field to the intermediate database;

[0266] Synchronize the data of the intermediate database to the date partition table of distributed storage, and establish partition storage for the date partition table based on the synchronization date field;

[0267] Receive input content of the user identification field, the target data table name field, and the data type field on the report page;

[0268] Determine the data table associated field according to the data type field, match the data value field of the date partition table with the data table associated field based on the target data table name field, and generate a matching result set;

[0269] Submitting a security sign-off including the user identification field and the matching result set through a report portal, and executing a security approval process based on the security sign-off;

[0270] When the security approval process is completed, the matching result set is sent to the user terminal according to the binding relationship between the user identification field in the authority table and the name field of the target data table.

[0271] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0272] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0273] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0274] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A data matching method based on security approval, characterized in that: The following steps are involved: Create a user data table and set access authorization for the user data table; Identify the authorized user data table through the permission table, and insert the data in the user data table into the collection data table; Setting a scheduled task to synchronize the data in the collected data table to an intermediate database, and adding a source table name field and a synchronization date field to the intermediate database; Synchronize the data of the intermediate database to the date partition table of distributed storage, and establish partition storage for the date partition table based on the synchronization date field; Receive input content of the user identification field, the target data table name field, and the data type field on the report page; Determine the data table associated field according to the data type field, match the data value field of the date partition table with the data table associated field based on the target data table name field, and generate a matching result set; Submitting a security sign-off including the user identification field and the matching result set through a report portal, and executing a security approval process based on the security sign-off; When the security approval process is completed, the matching result set is sent to the user terminal according to the binding relationship between the user identification field in the authority table and the name field of the target data table.

2. The data matching method based on security approval according to claim 1, characterized in that: Create a user data table and set access authorization for the user data table, including: Receive a data structure definition including a data sequence number field, a data type field, a data value field, and a user identification field; Create a user data table based on the data structure definition, and write the data sequence number field, data type field, data value field and user identification field into the user data table structure information according to a preset field order; Generate a user data table name field corresponding to the user data table according to the user identification field and a preset naming strategy; Verify whether the data sequence number field in the user data table is a numeric field, whether the data type field is within a fixed value range, whether the data value field is a character field, and whether the user identification field complies with a preset format strategy; When the field type or field order verification of the data sequence number field, data type field, data value field, or user identification field fails, the user data table creation operation is terminated and an error log containing the error type identifier is generated; When the user data table is created, the binding relationship between the user data table name field and the user identification field is written into the permission table; Granting the user account of the collection user the permission to read the user data table and the permission to write the collection data table, generating a unique access credential number and establishing an association with the user identification field; The unique access credential number is returned to the user terminal, and an access control policy is configured to restrict access to the user data table to only requests carrying a valid access credential number.

3. The data matching method based on security approval according to claim 1, characterized in that: Identifying the authorized user data table through the permission table and inserting the data in the user data table into the collection data table includes: Query the user data table name field bound to the current user ID field in the permission table, and generate a list of user data tables that the current user has authorized based on the user data table name field; Verify whether each user data table name field in the user data table list exists in the database table directory; When the user data table name field exists in the database table directory, data records are read from the user data table according to the preset paging size; Perform field-level format conversion on each data record read; Insert the format-converted data records into the collection data table in batches, and generate a data insertion transaction log when each batch of data records is inserted; When a data record insertion failure is detected, the unfinished batch of data record insertion operations is rolled back according to the data insertion transaction log and an alarm notification is triggered.

4. The data matching method based on security approval according to claim 1, characterized in that: Synchronizing the data of the intermediate database to a date partition table of distributed storage, and establishing partition storage for the date partition table based on the synchronization date field, including: Extract the user data table name field, data value field and synchronization date field from the intermediate database, and serialize them into column storage structure data; Generate a partition identifier according to the date value of the synchronization date field, wherein the partition identifier is organized into a year directory, a month directory, and a day directory in a hierarchical structure; Write the serialized columnar storage structure data into the date partition table of distributed storage according to the preset batch size. Generate an independent data file for each batch of columnar storage structure data and store it in the directory identified by the corresponding partition. Performing a data integrity check on the written data file to verify whether the total number of records in the user data table name field, data value field, and synchronization date field in the date partition table is consistent with the total number of synchronized records in the intermediate database; if the verification passes, marking the data synchronization status as completed; According to the preset data retention period policy, the year directory, month directory, date directory and data files stored in the corresponding directories in the date partition table that exceed the retention period are deleted.

5. The data matching method based on security approval according to claim 1, characterized in that: Determining a data table associated field according to the data type field, matching the data value field of the date partition table with the data table associated field based on the target data table name field, and generating a matching result set, including: Determine the name of the base table to be associated and the associated field in the corresponding base table according to the value of the data type field; According to the target data table name field, filter out data records whose source table name field is equal to the target data table name field from the date partition table; Extracting the data value field corresponding to the data type field from the filtered data records to generate a data set to be matched; Based on the name of the basic table and the associated fields, query the target database for a complete list of values ​​for the associated fields in the basic table; Matching the data value field in the to-be-matched data set with the complete value list of the associated field, and filtering out the target data value field that exists in the complete value list; A matching result set is generated based on the target data table name field, target data value field and associated fields.

6. The data matching method based on security approval according to claim 1, characterized in that: After determining the data table associated fields according to the data type field, matching the data value fields of the date partition table with the data table associated fields based on the target data table name field, and generating a matching result set, the method further includes: Receive the organizational unit code of the user terminal on the report page, and generate an authorized organizational unit list based on the organizational unit code and the preset organizational structure tree relationship; Extracting the data record to be screened that contains the organizational unit field associated with the authority from the matching result set; Based on the authorized organizational unit list, target data records whose values ​​of the organizational unit fields belong to the authorized organizational unit list are screened out from the data records to be screened, and a final matching result set is generated based on the target data records.

7. The data matching method based on security approval according to claim 1, characterized in that: Submitting a security sign-off including the user identification field and the matching result set through a report portal, and executing a security approval process based on the security sign-off, including: Generate a security report request in the report portal that includes the user identification field, the matching result set, and the report submission timestamp; Submit the security sign-off request to a preset security approval queue, which is associated with multiple levels of approval nodes and approval role permissions; The approval policy module determines the approval path of the security sign-on request based on the permission attribute of the user identification field, the preset approval policy in the approval policy library, and the metadata of the security sign-on request, and triggers the corresponding approval node to generate an approval task; The operation log of each approval node is recorded in the approval process. When all approval nodes complete the corresponding approval tasks and the approval status is passed, an approval pass mark is generated.

8. A data matching device based on security approval, characterized in that: The data matching device based on security approval includes: A user data table configuration module is used to create a user data table and set access authorization for the user data table; A data collection and processing module is used to identify the authorized user data table through the permission table and insert the data in the user data table into the collection data table; An intermediate data synchronization module is used to set a scheduled task to synchronize the data in the collected data table to the intermediate database, and to add a source table name field and a synchronization date field to the intermediate database; A distributed storage management module, configured to synchronize the data in the intermediate database to a date partition table of distributed storage, and to establish partition storage for the date partition table based on the synchronization date field; The report input receiving module is used to receive the input content of the user identification field, the target data table name field and the data type field on the report page; a data matching execution module, configured to determine a data table associated field according to the data type field, match the data value field of the date partition table with the data table associated field based on the target data table name field, and generate a matching result set; A security sign-on approval module, configured to submit a security sign-on containing the user identification field and the matching result set through a report portal, and execute a security approval process based on the security sign-on; The matching result sending module is used to send the matching result set to the user terminal according to the binding relationship between the user identification field in the authority table and the name field of the target data table when the security approval process is completed.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a data matching program based on security approval stored in the memory and capable of running on the processor. When the data matching program based on security approval is executed by the processor, the steps of the data matching method based on security approval as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a data matching program based on security approval, and when the data matching program based on security approval is executed by the processor, the steps of the data matching method based on security approval according to any one of claims 1 to 7 are implemented.

Images