A method and system for secure processing and transmission of massive amounts of data based on business processes

By using static code analysis and machine learning to identify sensitive data, and combining irreversible hashing and reversible encryption algorithms, the problem of inaccurate data query range acquisition and low efficiency of large-scale data transmission in distributed storage systems is solved, thus achieving secure and efficient transmission of massive amounts of data.

CN123123018APending Publication Date: 2026-11-17NANJING FENGTAI INTERNET TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611583372.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-13
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies struggle to automatically and accurately determine the scope of data queries in distributed storage systems, leading to a high risk of sensitive information leakage and low efficiency in large-scale data transmission.

Method used

The system obtains the data source location identifier and query condition parameters through static code parsing, identifies sensitive data using a machine learning classification model, processes sensitive data using irreversible hashing and reversible cryptographic encryption algorithms, and performs fragmentation and asynchronous transmission when the data exceeds the capacity of a single transmission.

Benefits of technology

It enables the efficient and reliable transmission of massive amounts of business data while ensuring the security of sensitive information, and ensures the security and privacy protection of data during storage, transmission and display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123018A_ABST
    Figure CN123123018A_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for secure processing and transmission of massive amounts of data based on business processes. It locates the data source and extracts massive business records through static code parsing, performs sensitivity feature assessment on data fields to identify sensitive and non-sensitive business data, de-identifies and anonymizes identity-related information, and performs confidentiality conversion on financial asset information. The processed business data is then used to construct a target query response dataset according to a predefined structure template. When the data volume exceeds the single transmission limit, it automatically performs block slicing processing and sends the slicing data units to the query processing terminal for integrity verification through an asynchronous transmission mechanism that supports multiplexing. This achieves efficient and reliable transmission of massive amounts of business data while ensuring the security of sensitive information, solving the technical problems of privacy leakage risks and low efficiency of large-scale data transmission faced by sensitive business data in the query response process in a distributed environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for secure processing and transmission of massive amounts of data based on business processes. Background Technology

[0002] In recent years, as businesses have expanded, online service platforms have undertaken an increasing number of core business processing tasks, resulting in a rapid increase in the amount of data generated during operations. This data is typically stored in distributed data storage systems and involves various fields such as user identification information and transaction logs, including sensitive information such as ID card numbers, mobile phone numbers, and account balances. When business personnel need to perform statistical analysis and queries on massive amounts of data, they must ensure both fast query response times and prevent the leakage of sensitive information during storage, transmission, and display.

[0003] In existing technologies, the data query scope corresponding to business processes often relies on manual sorting and configuration based on functional module code. It is difficult to automatically and accurately obtain data query scope configuration information, including data source location identifiers and query condition parameters, from the functional module code. This can easily lead to omissions or expansions in the query scope, affecting the accuracy and efficiency of subsequent data acquisition. At the same time, the field formats and attributions of massive data records are inconsistent, and the identification of sensitive fields often relies on fixed rules or manual experience, which is difficult to adapt to changes in field types and business evolution, and is prone to missed or false positives.

[0004] Regarding sensitive data processing, existing solutions often employ uniform desensitization or encryption methods. If only reversible encryption is used for identity verification information, the risk of data recovery and leakage remains. If irreversible desensitization is used for financial asset information, data availability may be compromised, impacting business queries and subsequent processing. Furthermore, when the query response dataset is large and exceeds the single transmission capacity limit, direct transmission can easily cause timeouts, failures, or interruptions, while simple fragmentation makes it difficult to guarantee the integrity of each fragment and ensure receipt confirmation.

[0005] Therefore, how to automatically obtain the accurate data query range from the functional module code corresponding to the business process, perform field-level parsing and sensitivity assessment on the massive raw business data in distributed storage, and carry out differentiated security processing for identity recognition information and financial asset information, while achieving reliable fragmentation, asynchronous transmission and integrity verification when the response dataset exceeds the single transmission capacity, has become an urgent technical problem to be solved to improve the data query efficiency and security protection capabilities of the business system. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for secure processing and transmission of massive amounts of data based on business processes, so as to solve the problems mentioned in the background art.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] A method for secure processing and transmission of massive amounts of data based on business processes, comprising the following steps:

[0009] Obtain the functional module code corresponding to the business process, perform static code parsing on the functional module code, and obtain data query range configuration information including data source location identifier and query condition parameters;

[0010] Based on the data query range configuration information, a massive amount of data records related to the business process are obtained from the distributed data storage system. The massive amount of data records are parsed at the field level to extract the original business data containing user identification information and transaction flow information.

[0011] Based on the original business data, a machine learning classification model is used to evaluate the sensitivity features of the data fields of the original business data, and sensitive business data and non-sensitive business data are obtained according to the information type carried by the data fields.

[0012] Determine whether the field attributes of the sensitive business data belong to identity recognition information. If the field attributes of the sensitive business data belong to identity recognition information, then use an irreversible hash algorithm to perform one-way transformation processing on the sensitive business data to obtain de-identified and de-identified business data.

[0013] Determine whether the field attributes of the sensitive business data belong to financial asset information. If the field attributes of the sensitive business data belong to financial asset information, then use a reversible cryptographic encryption algorithm to perform ciphertext conversion processing on the sensitive business data to obtain confidential business data.

[0014] According to the data structure template predefined by the business system, de-identified business data, confidential business data and non-sensitive business data are sequentially concatenated to construct the target query response dataset;

[0015] Obtain the data size of the target query response dataset, determine whether the data size exceeds the single transmission capacity limit, and if the data size exceeds the single transmission capacity limit, then perform block slicing processing on the target query response dataset to obtain multiple independently transmitted fragmented data units.

[0016] For multiple independently transmitted fragmented data units, an asynchronous transmission mechanism supporting multiplexing is used to send them to the query processing terminal, and the query processing terminal returns a data reception confirmation identifier after performing integrity verification on the multiple independently transmitted fragmented data units.

[0017] Furthermore, the specific implementation process of obtaining massive data records related to the business process from the distributed data storage system according to the data query range configuration information, performing field-level parsing on the massive data records, and extracting original business data containing user identification information and transaction flow information includes:

[0018] Parse the data source location identifier in the data query range configuration information to locate the target data node in the distributed data storage system;

[0019] Based on the query condition parameters in the data query scope configuration information, extract massive data records from the target data node;

[0020] The massive data records are structurally decomposed to obtain each data field of the massive data records;

[0021] Identify the user identification information and transaction flow information in each of the data fields, and combine the user identification information and transaction flow information to generate the original business data.

[0022] Furthermore, the specific implementation process of evaluating the sensitivity features of the data fields of the original business data using a machine learning classification model, and classifying the data fields into sensitive and non-sensitive business data according to the information type they carry, includes:

[0023] Obtain the total number of fields in the original business data, extract each data field from the original business data, and calculate the sensitivity weight coefficient corresponding to each data field.

[0024] Each data field is input into a preset machine learning classification model, which then classifies and determines the data field and outputs the classification result.

[0025] By combining the sensitivity weight coefficient and the classification judgment result, the comprehensive sensitivity score of the original business data is calculated;

[0026] A Boolean function is used to determine whether each data field is sensitive information. If the Boolean function outputs true, the information type carried by the data field is determined to be sensitive information and is classified as sensitive business data.

[0027] If the Boolean function outputs false, the information type carried by the data field is determined to be non-sensitive information, and it is classified as non-sensitive business data.

[0028] Furthermore, the specific implementation process for obtaining de-identified and anonymized business data includes:

[0029] Extract the field attribute types of sensitive business data and compare them with the preset identity recognition information feature library;

[0030] If the field attribute type matches the identity recognition information feature library, it is determined that the sensitive business data contains identity recognition information.

[0031] An irreversible hash algorithm is invoked to perform a hash operation on the sensitive business data containing the aforementioned identity recognition information, generating a fixed-length hash value, which is then stored as de-identified and anonymized business data.

[0032] Furthermore, the specific implementation process for obtaining confidential business data includes:

[0033] Extract the field attribute types of sensitive business data and compare them with the preset financial asset information feature database;

[0034] If the field attribute type matches the financial asset information feature library, it is determined that the sensitive business data contains financial asset information.

[0035] Obtain the pre-allocated encryption key, call the reversible cryptographic encryption algorithm, and combine the encryption key to perform symmetric encryption operation on sensitive business data containing financial asset information to generate encrypted ciphertext;

[0036] The encrypted ciphertext is output as the confidential business data.

[0037] Furthermore, the specific implementation process for constructing the target query response dataset includes:

[0038] Parse the predefined data structure template of the business system, which includes the specified field arrangement order and data format specifications;

[0039] Extract anonymized business data, confidential business data, and non-sensitive business data according to the order of the fields described.

[0040] In accordance with data format specifications, the format alignment process is performed on de-identified business data, confidential business data, and non-sensitive business data.

[0041] The format-aligned, anonymized business data, confidential business data, and non-sensitive business data are combined to generate the target query response dataset.

[0042] Furthermore, the specific implementation process for obtaining multiple independently transmitted fragmented data units includes:

[0043] The total number of bytes in the target query response dataset is used as the data size.

[0044] Read the system's configured single transmission capacity limit and compare the data capacity size with the single transmission capacity limit;

[0045] If the data capacity exceeds the single transmission capacity limit, calculate the ratio of the data capacity to the single transmission capacity limit.

[0046] The comparison value is rounded up to determine the number of slices.

[0047] Calculate the data length of each shard based on the number of shards and the total length of the target query response dataset;

[0048] Based on the data length of each shard, the target query response dataset is sequentially truncated to generate multiple independently transmitted shard data units.

[0049] Furthermore, for multiple independently transmitted fragmented data units, an asynchronous transmission mechanism supporting multiplexing is used to send them to the query processing terminal. The specific implementation process of receiving the data reception confirmation identifier returned by the query processing terminal after performing integrity verification on the multiple independently transmitted fragmented data units includes:

[0050] Initialize the window size and initial sliding position of the sliding window control;

[0051] Obtain multiple independently transmitted fragmented data units, and allocate corresponding multistream transmission channels to the multiple independently transmitted fragmented data units through sliding window control to obtain fragmented packets to be sent with transmission channel identifiers;

[0052] Extract the header information and payload data of the fragmented packets to be sent, perform polynomial division on the header information and payload data, and use the remainder in the result as the check code;

[0053] For a fragment packet to be sent with a channel identifier, a cyclic redundancy check algorithm is used to calculate the checksum of the fragment packet to be sent, and an asynchronous stream data block carrying the checksum is obtained.

[0054] The asynchronous stream data block carrying the verification code is sent to the query processing terminal through an asynchronous transmission mechanism that supports multiplexing;

[0055] Receive the receive status bits for the asynchronous stream data block returned by the query processing terminal;

[0056] Determine if the receiving status bit is in the receiving completed state. If the receiving status bit is in the receiving completed state, obtain the integrity identifier generated after comparing the checksum extracted from the asynchronous stream data block by the query processing terminal.

[0057] The integrity identifier is used to determine whether the asynchronous stream data block has passed the integrity check. If the integrity identifier indicates that the integrity check has passed, a data reception acknowledgment code for the corresponding asynchronous stream data block is generated and used as the data reception acknowledgment identifier.

[0058] Obtain the data reception confirmation code, update the sliding position of the sliding window control based on the data reception confirmation code, and determine the next fragment to be sent.

[0059] A system for secure processing and transmission of massive amounts of data based on business processes, the system comprising:

[0060] The original business data extraction module is used to obtain the functional module code corresponding to the business process, perform static code parsing on the functional module code to obtain data query range configuration information containing data source location identifier and query condition parameters; according to the data query range configuration information, it obtains massive data records related to the business process from the distributed data storage system, performs field-level parsing on the massive data records, and extracts original business data containing user identification information and transaction flow information.

[0061] The sensitivity feature assessment module is used to assess the sensitivity features of the data fields of the original business data using a machine learning classification model, and to classify the data fields into sensitive business data and non-sensitive business data according to the information type they carry.

[0062] The de-identified business data generation module is used to determine whether the field attributes of the sensitive business data belong to identity recognition information. If the field attributes of the sensitive business data belong to identity recognition information, an irreversible hash algorithm is used to perform one-way transformation processing on the sensitive business data to obtain de-identified business data.

[0063] The confidential business data generation module is used to determine whether the field attributes of the sensitive business data belong to financial asset information. If the field attributes of the sensitive business data belong to financial asset information, a reversible cryptographic encryption algorithm is used to perform ciphertext conversion processing on the sensitive business data to obtain confidential business data.

[0064] The target query response dataset construction module is used to construct the target query response dataset by sequentially concatenating anonymized business data, confidential business data, and non-sensitive business data according to the data structure template predefined by the business system.

[0065] The data fragment generation module is used to obtain the data size of the target query response dataset, determine whether the data size exceeds the single transmission capacity limit, and if the data size exceeds the single transmission capacity limit, then the target query response dataset is divided into blocks and sliced ​​to obtain multiple independently transmitted data fragments.

[0066] The data reception confirmation identifier receiving module is used to send multiple independently transmitted fragmented data units to the query processing terminal using an asynchronous transmission mechanism that supports multiplexing, and to receive the data reception confirmation identifier returned by the query processing terminal after performing integrity verification on the multiple independently transmitted fragmented data units.

[0067] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention provides a method and system for secure processing and transmission of massive amounts of data based on business processes, solving the technical challenges of privacy leakage risks and low efficiency in large-scale data transmission faced by sensitive business data in a distributed environment during query response. This method locates the data source and extracts massive business records through static code parsing, uses a machine learning classification model to assess the sensitivity features of data fields, automatically identifies sensitive and non-sensitive business data, employs irreversible hashing algorithms for de-identification and anonymization of identity-related information, and uses reversible cryptographic encryption algorithms for confidentiality conversion of financial asset information. The processed anonymized data, confidential data, and non-sensitive data are then used to construct a target query response dataset according to a predefined structure template. When the data volume exceeds the single transmission limit, automatic chunking and slicing processing is performed, and the slicing data units are sent to the query processing terminal for integrity verification through an asynchronous transmission mechanism supporting multiplexing. This achieves efficient and reliable transmission of massive amounts of business data while ensuring the security of sensitive information. It effectively protects data security and privacy while processing large amounts of data, ensuring the security and reliability of data during storage, transmission, and display. Attached Figure Description

[0068] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0069] Figure 1 This is a schematic diagram illustrating the steps of a method for secure processing and transmission of massive amounts of data based on business processes, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the processing flow of sensitive business data in an embodiment of the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] Please see Figures 1-2 In this first embodiment, a method for secure processing and transmission of massive amounts of data based on business processes is provided. This method includes the following steps:

[0072] S101: Obtain the functional module code corresponding to the business process, perform static code parsing on the functional module code, and obtain data query range configuration information including data source location identifier and query condition parameters;

[0073] S102: Based on the data query range configuration information, obtain massive data records related to the business process from the distributed data storage system, perform field-level parsing on the massive data records, and extract the original business data containing user identification information and transaction flow information;

[0074] S103: Based on the original business data, a machine learning classification model is used to evaluate the sensitivity features of the data fields of the original business data, and sensitive business data and non-sensitive business data are obtained according to the information type carried by the data fields.

[0075] S104: Determine whether the field attributes of the sensitive business data belong to identity recognition information. If the field attributes of the sensitive business data belong to identity recognition information, then use an irreversible hash algorithm to perform one-way transformation processing on the sensitive business data to obtain de-identified and de-identified business data.

[0076] S105: Determine whether the field attributes of the sensitive business data belong to financial asset information. If the field attributes of the sensitive business data belong to financial asset information, then use a reversible cryptographic encryption algorithm to perform ciphertext conversion processing on the sensitive business data to obtain confidential business data.

[0077] S106: According to the data structure template predefined by the business system, sequentially concatenate anonymized business data, confidential business data, and non-sensitive business data to construct the target query response dataset;

[0078] S107: Obtain the data size of the target query response dataset, determine whether the data size exceeds the single transmission capacity limit, and if the data size exceeds the single transmission capacity limit, then perform block slicing processing on the target query response dataset to obtain multiple independently transmitted fragmented data units.

[0079] S108: For multiple independently transmitted fragmented data units, an asynchronous transmission mechanism supporting multiplexing is used to send them to the query processing terminal, and the query processing terminal returns a data reception confirmation identifier after performing integrity verification on the multiple independently transmitted fragmented data units.

[0080] Specifically, based on the data query range configuration information, the process of retrieving massive data records related to the business process from the distributed data storage system, performing field-level parsing on the massive data records, and extracting raw business data containing user identification information and transaction flow information includes:

[0081] Parse the data source location identifier in the data query range configuration information to locate the target data node in the distributed data storage system;

[0082] Based on the query condition parameters in the data query scope configuration information, extract massive data records from the target data node;

[0083] The massive data records are structurally decomposed to obtain each data field of the massive data records;

[0084] Massive data records can be preprocessed into smaller, single-machine-processable fragments through methods such as hash partitioning, consistent hashing, range partitioning, and Bloom filter deduplication. Different formats of raw records can be identified and segmented into individual complete data entries using methods such as SAX streaming XML parsing, incremental JSON parsing, regular expression boundary matching, and Lxml HTML parse tree construction. Finally, all independent and standardized data fields can be extracted from a single record using methods such as regular expression grouping capture, CSS / XPath path extraction, recursive sub-tag matching, and dynamic field mapping during read-time modeling.

[0085] Identify the user identification information and transaction flow information in each of the data fields, and combine the user identification information and transaction flow information to generate the original business data.

[0086] Specifically, the implementation process of evaluating the sensitivity features of data fields in the original business data using a machine learning classification model, and classifying sensitive and non-sensitive business data according to the information type carried by the data fields, includes:

[0087] Obtain the total number of fields in the original business data, extract each data field from the original business data, and calculate the sensitivity weight coefficient corresponding to each data field.

[0088] Specifically, the sensitivity weight coefficient of each data field can be calculated using methods such as basic subjective weighting, AHP (Analytic Hierarchy Process), and entropy method; no restrictions are imposed here.

[0089] Each data field is input into a preset machine learning classification model, which then classifies and determines the data field and outputs the classification result.

[0090] Combining the sensitivity weight coefficient and the classification result, the comprehensive sensitivity score of the original business data is calculated using the following formula:

[0091] ;

[0092] ;

[0093] Where M represents the total number of fields in the original business data. This represents the i-th data field. This represents the sensitivity weight coefficient of the i-th field. This indicates the machine learning classification model's response to data fields. The classification results Indicates the data field used for judgment A Boolean function to determine whether the information is sensitive. This represents the overall sensitivity score of the original business data;

[0094] A Boolean function is used to determine whether each data field is sensitive information. If the Boolean function outputs true, the information type carried by the data field is determined to be sensitive information and is classified as sensitive business data.

[0095] If the Boolean function outputs false, the information type carried by the data field is determined to be non-sensitive information and classified as non-sensitive business data.

[0096] The specific implementation process for obtaining de-identified and anonymized business data includes:

[0097] Extract the field attribute types of sensitive business data and compare them with the preset identity recognition information feature library;

[0098] If the field attribute type matches the identity recognition information feature library, it is determined that the sensitive business data contains identity recognition information.

[0099] An irreversible hash algorithm is invoked to perform a hash operation on the sensitive business data containing the identity recognition information, generating a hash value of fixed length, and storing the hash value as de-identified and de-identified business data.

[0100] Specifically, the specific implementation process for obtaining confidential business data includes:

[0101] Extract the field attribute types of sensitive business data and compare them with the preset financial asset information feature database;

[0102] If the field attribute type matches the financial asset information feature library, it is determined that the sensitive business data contains financial asset information.

[0103] Obtain the pre-allocated encryption key, call the reversible cryptographic encryption algorithm, and combine the encryption key to perform symmetric encryption operation on sensitive business data containing financial asset information to generate encrypted ciphertext;

[0104] The encrypted ciphertext is output as the confidential business data.

[0105] The sensitive business data is processed using the following formula:

[0106] ;

[0107] in, This indicates sensitive business data. This represents the processed and de-identified business data. Type() is a function to determine the type of data field attributes. Identity represents identity recognition information, Financial represents financial asset information, Hash() is an irreversible hash algorithm used to generate a unique digital fingerprint for sensitive business data, supporting various data security and verification scenarios, Encrypt() is a reversible cryptographic encryption algorithm, and K represents the encryption key.

[0108] Specifically, the specific implementation process for constructing the target query response dataset includes:

[0109] Parse the predefined data structure template of the business system, the data structure template including the specified field arrangement order and data format specifications;

[0110] Extract anonymized business data, confidential business data, and non-sensitive business data according to the order of the fields described.

[0111] In accordance with data format specifications, the format alignment process is performed on de-identified business data, confidential business data, and non-sensitive business data.

[0112] The format-aligned, anonymized business data, confidential business data, and non-sensitive business data are combined to generate the target query response dataset.

[0113] Based on data format specifications, define fully aligned global field standards for three types of data:

[0114] Unified field naming: For data of the same source, including data that is de-identified, confidential, and non-sensitive, the field name must be completely consistent. For example, the user mobile phone number field in the three types of data should be named user_phone to avoid subsequent association failures caused by different names for the same field.

[0115] Unified data types: Force data type alignment for fields of the same origin. For example, the user ID field is uniformly defined as BIGINT for all three types of data, and the time field is uniformly defined as DATETIME(6), which completely eliminates the problem of data type incompatibility across different security levels.

[0116] A new security level identifier field has been added: a security_level field has been added to all three types of business data, with fixed values ​​of 1=non-sensitive, 2=de-identified, and 3=confidential, serving as a unified entry identifier for subsequent permission routing and security verification.

[0117] Based on the security characteristics of the three types of data, format conversion is performed separately to align the formats without compromising security attributes:

[0118] Non-sensitive business data undergoes full format cleaning, filling in missing fields, correcting illegal characters, and mapping all data directly to the global standard schema without additional security constraints.

[0119] While preserving the irreversibility of the data masking results, the masked field lengths, character sets, and formatting rules are aligned with global standards, such as the masked mobile phone numbers. The characters are uniformly aligned to an 11-character length, which is completely consistent with the field length specification of the original mobile phone number.

[0120] Confidential business data is format-mapped under transparent encryption. The encrypted ciphertext fields are uniformly aligned to a fixed-length Base64 encoded string. The field names and field types are completely consistent with the other two types of fields of the same origin. The original plaintext is never decrypted throughout the process, thus avoiding the risk of confidential data leakage.

[0121] Unified underlying encoding and storage rules across the entire chain eliminate format differences at the physical level.

[0122] The unified character encoding completely converts all three types of business data to UTF-8 encoding, thus resolving the garbled text problem caused by the mixing of multiple encodings such as GBK and ISO-8859-1.

[0123] Unify the time / numerical format. All time fields are uniformly aligned to the format yyyy-MM-dd HH:mm:ss.SSS. Amounts are uniformly aligned to DECIMAL(18,2). The rules for thousands place and decimal place are uniform globally.

[0124] Unified storage partitioning: Three types of business data from the same source are distributed and partitioned according to the same partition key (such as the dt date field), which ensures that the data routing rules are completely consistent and the query performance is not affected when performing cross-security level related queries.

[0125] An automated verification mechanism ensures that the format alignment is 100% compliant with the specifications.

[0126] First, batch verification is performed. Through a distributed verification task, the field names, types, lengths, and formats of the three types of data from the same source are checked row by row to see if they match completely, and all dirty data with misalignment is marked.

[0127] Next, permission isolation verification confirms that the encrypted fields of confidential data can only be accessed by authorized roles, and that the irreversible nature of de-identified data has not been destroyed by the format alignment process, thus avoiding security vulnerabilities.

[0128] Finally, an automatic reflow is performed for any errors. All data that fails to align is automatically reflowed to the preprocessing stage for reprocessing until it fully conforms to the global format specification before it is allowed to be written to the unified data layer.

[0129] Specifically, the specific implementation process of obtaining multiple independently transmitted fragmented data units includes:

[0130] The total number of bytes in the target query response dataset is used as the data size.

[0131] Read the system's configured single transmission capacity limit and compare the data capacity size with the single transmission capacity limit;

[0132] If the data capacity exceeds the single transmission capacity limit, calculate the ratio of the data capacity to the single transmission capacity limit.

[0133] The comparison value is rounded up to determine the number of slices.

[0134] The data length of each shard is calculated based on the number of shards and the total length of the target query response dataset. The calculation formula is as follows:

[0135] ;

[0136] ;

[0137] in, This indicates the data size of the target query response dataset. Indicates the maximum capacity of a single transmission. This indicates the number of shards to be split, where L represents the data length of each shard. Rounding up indicates that the number of shards must be an integer, while rounding down indicates the integer part of the shard length.

[0138] Based on the data length of each shard, the target query response dataset is sequentially truncated to generate multiple independently transmitted shard data units.

[0139] Specifically, the implementation process of sending multiple independently transmitted fragmented data units to the query processing terminal using an asynchronous transmission mechanism that supports multiplexing, and receiving the data reception confirmation identifier returned by the query processing terminal after performing integrity verification on the multiple independently transmitted fragmented data units, includes:

[0140] Initialize the window size and initial sliding position of the sliding window control;

[0141] Obtain multiple independently transmitted fragmented data units, and allocate corresponding multistream transmission channels to the multiple independently transmitted fragmented data units through sliding window control to obtain fragmented packets to be sent with transmission channel identifiers;

[0142] Extract the header information and payload data of the fragmented packets to be sent, perform polynomial division on the header information and payload data, and use the remainder in the result as the check code;

[0143] For a fragment packet to be sent with a channel identifier, a cyclic redundancy check algorithm is used to calculate the checksum of the fragment packet to be sent, and an asynchronous stream data block carrying the checksum is obtained.

[0144] The packet header includes at least the channel identifier, fragment sequence number, source terminal identifier, fragment length, and total number of fragments in the data block to which it belongs; wherein the channel identifier is used to distinguish different logical data streams in a multiplexed link.

[0145] For each packet fragment to be transmitted, its header information and payload data are extracted and concatenated in a fixed order to form a bit stream to be verified, i.e., header first, payload second, and the bit order within the bit stream follows the most significant bit first (MSBfirst) order, with bytes arranged in ascending order of address. If the header contains fields that change dynamically with the transmission time (such as timestamps), these fields are not included in the verification calculation and are removed from the bit stream to be verified before the verification calculation, only being sent along with the packet as is after the checksum is generated.

[0146] For fragmented packets to be transmitted that bear a channel identifier, a cyclic redundancy check (CRC) algorithm is used to calculate the checksum. The specific steps are as follows:

[0147] A pre-defined generator polynomial G(x) with order r is chosen, corresponding to a checksum length of r bits. In one implementation, CRC-32 is used, with the generator polynomial being... Its bit sequence is 0x04C11DB7, r=32.

[0148] The check register is initialized to the initial value INIT, and in one implementation, INIT=0xFFFFFFFF.

[0149] The bit stream to be checked is sent to the CRC calculation engine byte by byte and processed according to the agreed bit order; in one implementation, the input bytes are not bit reflected (refin=false).

[0150] Perform modulo-2 division on the entire bit stream to be checked, that is, perform polynomial division on the bit sequence obtained by "padding r zeros after the bit stream to be checked" with G(x) as the divisor, and the remainder is the intermediate check value.

[0151] The intermediate check value is processed according to the convention and then output: In one implementation, the output is not bit-reflected (refout=false) and is XORed with the output XOR mask XOROUT, XOROUT=0xFFFFFFFF. The result is the final check code, which is 32 bits (4 bytes) long.

[0152] The channel identifier is part of the packet header and is already included in the bit stream to be checked. Therefore, its value will naturally be reflected in the checksum as the packet differs. For fragmented packets that do not carry the channel identifier, the checksum is still calculated using the same polynomial and the same set of parameters to ensure that the receiving end can use the same set of logic to check all fragmented packets.

[0153] The calculated checksum is embedded into the fragment packet to obtain an asynchronous stream data block carrying the checksum. In one implementation, the checksum is appended as 4 bytes to the end of the payload, with big-endian byte order (most significant byte first). The asynchronous stream data block includes at least a stream type identifier, the corresponding channel identifier, the block sequence number, the total block length, the original fragment packet (packet header + payload), and a 32-bit checksum.

[0154] A fragment packet to be sent is encapsulated into an asynchronous stream data block, and the two correspond one-to-one without merging multiple packets; the boundary between asynchronous stream data blocks is defined by the total block length field in the block header.

[0155] In this invention, CRC-32 (0x04C11DB7, INIT=0xFFFFFFFF, XOROUT=0xFFFFFFFF) is used as an example implementation, which can be replaced by other polynomials such as CRC-16 and CRC-CCITT.

[0156] The asynchronous stream data block carrying the verification code is sent to the query processing terminal through an asynchronous transmission mechanism that supports multiplexing;

[0157] The transmitting end maintains virtual channels that correspond one-to-one with each transport channel identifier, and each virtual channel is strictly queued according to the block sequence number in an ascending order.

[0158] On the physical link, each non-empty channel is polled in a packet-level time-division multiplexing manner, and asynchronous stream data blocks are sent to the link in sequence; blocks within the same transmission channel maintain the sequence number order, but the order between different transmission channels is not guaranteed;

[0159] The asynchronous transmission mechanism includes flow control and timeout retransmission: the receiver acknowledges the transmission channel, and the sender retransmits the unacknowledged block after the agreed timeout; the retransmitted block uses the already calculated and encapsulated checksum, without recalculation.

[0160] Receive the receive status bits for the asynchronous stream data block returned by the query processing terminal;

[0161] Determine if the receiving status bit is in the receiving completed state. If the receiving status bit is in the receiving completed state, obtain the integrity identifier generated after comparing the checksum extracted from the asynchronous stream data block by the query processing terminal.

[0162] The integrity identifier is used to determine whether the asynchronous stream data block has passed the integrity check. If the integrity identifier indicates that the integrity check has passed, a data reception acknowledgment code for the corresponding asynchronous stream data block is generated and used as the data reception acknowledgment identifier.

[0163] Obtain the data reception confirmation code, update the sliding position of the sliding window control based on the data reception confirmation code, and determine the next fragment to be sent.

[0164] A system for secure processing and transmission of massive amounts of data based on business processes, the system comprising:

[0165] The original business data extraction module is used to obtain the functional module code corresponding to the business process, perform static code parsing on the functional module code to obtain data query range configuration information containing data source location identifier and query condition parameters; according to the data query range configuration information, it obtains massive data records related to the business process from the distributed data storage system, performs field-level parsing on the massive data records, and extracts original business data containing user identification information and transaction flow information.

[0166] The sensitivity feature assessment module is used to assess the sensitivity features of the data fields of the original business data using a machine learning classification model, and to classify the data fields into sensitive business data and non-sensitive business data according to the information type they carry.

[0167] The de-identified business data generation module is used to determine whether the field attributes of the sensitive business data belong to identity recognition information. If the field attributes of the sensitive business data belong to identity recognition information, an irreversible hash algorithm is used to perform one-way transformation processing on the sensitive business data to obtain de-identified business data.

[0168] The confidential business data generation module is used to determine whether the field attributes of the sensitive business data belong to financial asset information. If the field attributes of the sensitive business data belong to financial asset information, a reversible cryptographic encryption algorithm is used to perform ciphertext conversion processing on the sensitive business data to obtain confidential business data.

[0169] The target query response dataset construction module is used to construct the target query response dataset by sequentially concatenating anonymized business data, confidential business data, and non-sensitive business data according to the data structure template predefined by the business system.

[0170] The data fragment generation module is used to obtain the data size of the target query response dataset, determine whether the data size exceeds the single transmission capacity limit, and if the data size exceeds the single transmission capacity limit, then the target query response dataset is divided into blocks and sliced ​​to obtain multiple independently transmitted data fragments.

[0171] The data reception confirmation identifier receiving module is used to send multiple independently transmitted fragmented data units to the query processing terminal using an asynchronous transmission mechanism that supports multiplexing, and to receive the data reception confirmation identifier returned by the query processing terminal after performing integrity verification on the multiple independently transmitted fragmented data units.

[0172] For example, the system first slices the original dataset, which has a total size of 500 megabytes, into 50 independent data fragments, each 10 megabytes in size. Then, the sending end initiates a multiplexed asynchronous transmission mechanism based on a flow control transmission protocol, establishing 10 parallel transmission channels within a single network connection. These 50 fragments are dynamically allocated to these ten channels through weighted round-robin scheduling, with each channel asynchronously sending 5 fragments. During transmission, subsequent fragments can be sent continuously without waiting for a response from the previous fragment, thus improving network bandwidth utilization. Upon receiving these fragments, the query processing terminal immediately triggers a Cyclic Redundancy Check (CRC) 32 algorithm to perform integrity calculations on each fragment. The calculated checksum is rigorously compared with the original checksum carried in the fragment header. If the comparison matches, the terminal generates a 16-bit binary data reception acknowledgment identifier containing the fragment sequence number and a timestamp. The sending end continuously listens to a specific callback port and asynchronously receives 50 acknowledgment flags within a 200-millisecond timeout window. Once it is detected that the acknowledgment flag of a certain fragment has not arrived on time, the sending end will trigger a selective retransmission mechanism, only re-entering the missing specific fragment into the multiplexed channel for secondary transmission, until the acknowledgment flags of all 50 fragments have been successfully received and synchronously updated to the underlying state database.

[0173] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0174] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for secure processing and transmission of massive amounts of data based on business processes, characterized in that, The method includes the following steps: Obtain the functional module code corresponding to the business process, perform static code parsing on the functional module code, and obtain the data query range configuration information including the data source location identifier and query condition parameters; Based on the data query range configuration information, a massive amount of data records related to the business process are obtained from the distributed data storage system. The massive amount of data records are parsed at the field level to extract the original business data containing user identification information and transaction flow information. Based on the original business data, a machine learning classification model is used to evaluate the sensitivity features of the data fields of the original business data, and sensitive business data and non-sensitive business data are obtained according to the information type carried by the data fields. Determine whether the field attributes of the sensitive business data belong to identity recognition information. If the field attributes of the sensitive business data belong to identity recognition information, then use an irreversible hash algorithm to perform one-way transformation processing on the sensitive business data to obtain de-identified and de-identified business data. Determine whether the field attributes of the sensitive business data belong to financial asset information. If the field attributes of the sensitive business data belong to financial asset information, then use a reversible cryptographic encryption algorithm to perform ciphertext conversion processing on the sensitive business data to obtain confidential business data. According to the data structure template predefined by the business system, de-identified business data, confidential business data and non-sensitive business data are sequentially concatenated to construct the target query response dataset; Obtain the data size of the target query response dataset, determine whether the data size exceeds the single transmission capacity limit, and if the data size exceeds the single transmission capacity limit, then perform block slicing processing on the target query response dataset to obtain multiple independently transmitted fragmented data units. For multiple independently transmitted fragmented data units, an asynchronous transmission mechanism supporting multiplexing is used to send them to the query processing terminal, and the query processing terminal returns a data reception confirmation identifier after performing integrity verification on the multiple independently transmitted fragmented data units.

2. The method for secure processing and transmission of massive amounts of data based on business processes according to claim 1, characterized in that, The specific implementation process of obtaining massive data records related to the business process from the distributed data storage system according to the data query range configuration information, performing field-level parsing on the massive data records, and extracting original business data containing user identification information and transaction flow information includes: Parse the data source location identifier in the data query range configuration information to locate the target data node in the distributed data storage system; Based on the query condition parameters in the data query scope configuration information, extract massive data records from the target data node; The massive data records are structurally decomposed to obtain each data field of the massive data records; Identify the user identification information and transaction flow information in each of the data fields, and combine the user identification information and transaction flow information to generate the original business data.

3. The method for secure processing and transmission of massive amounts of data based on business processes according to claim 1, characterized in that, The specific implementation process of evaluating the sensitivity features of data fields in the original business data using a machine learning classification model, and classifying sensitive and non-sensitive business data according to the information type carried by the data fields, includes: Obtain the total number of data fields in the original business data, extract each data field from the original business data, and calculate the sensitivity weight coefficient corresponding to each data field; Each data field is input into a preset machine learning classification model, which then classifies and determines the data field and outputs the classification result. By combining the sensitivity weight coefficient and the classification judgment result, the comprehensive sensitivity score of the original business data is calculated; A Boolean function is used to determine whether each data field is sensitive information. If the Boolean function outputs true, the information type carried by the data field is determined to be sensitive information and is classified as sensitive business data. If the Boolean function outputs false, the information type carried by the data field is determined to be non-sensitive information and classified as non-sensitive business data.

4. The method for secure processing and transmission of massive amounts of data based on business processes according to claim 1, characterized in that, The specific implementation process for obtaining de-identified and anonymized business data includes: Extract the field attribute types of sensitive business data and compare them with the preset identity recognition information feature library; If the field attribute type matches the identity recognition information feature library, it is determined that the sensitive business data contains identity recognition information. An irreversible hash algorithm is invoked to perform a hash operation on the sensitive business data containing the identity recognition information, generating a hash value of fixed length, and storing the hash value as de-identified and de-identified business data.

5. The method for secure processing and transmission of massive amounts of data based on business processes according to claim 1, characterized in that, The specific implementation process for obtaining confidential business data includes: Extract the field attribute types of sensitive business data and compare them with the preset financial asset information feature database; If the field attribute type matches the financial asset information feature library, it is determined that the sensitive business data contains financial asset information. Obtain the pre-allocated encryption key, call the reversible cryptographic encryption algorithm, and combine the encryption key to perform symmetric encryption operation on sensitive business data containing financial asset information to generate encrypted ciphertext; The encrypted ciphertext is output as the confidential business data.

6. The method for secure processing and transmission of massive amounts of data based on business processes according to claim 1, characterized in that, The specific implementation process for constructing the target query response dataset includes: Parse the predefined data structure template of the business system, which includes the specified field arrangement order and data format specifications; Extract anonymized business data, confidential business data, and non-sensitive business data according to the order of the fields described. In accordance with data format specifications, the format alignment process is performed on de-identified business data, confidential business data, and non-sensitive business data. The format-aligned, anonymized business data, confidential business data, and non-sensitive business data are combined to generate the target query response dataset.

7. The method for secure processing and transmission of massive amounts of data based on business processes according to claim 1, characterized in that, The specific implementation process of obtaining multiple independently transmitted fragmented data units includes: The total number of bytes in the target query response dataset is used as the data size. Read the system's configured single transmission capacity limit and compare the data capacity size with the single transmission capacity limit; If the data capacity exceeds the single transmission capacity limit, calculate the ratio of the data capacity to the single transmission capacity limit. The comparison value is rounded up to determine the number of slices. Calculate the data length of each shard based on the number of shards and the total length of the target query response dataset; Based on the data length of each shard, the target query response dataset is sequentially truncated to generate multiple independently transmitted shard data units.

8. The method for secure processing and transmission of massive amounts of data based on business processes according to claim 1, characterized in that, For multiple independently transmitted fragmented data units, an asynchronous transmission mechanism supporting multiplexing is used to send them to the query processing terminal. The specific implementation process of receiving the data reception confirmation identifier returned by the query processing terminal after performing integrity verification on the multiple independently transmitted fragmented data units includes: Initialize the window size and initial sliding position of the sliding window control; Obtain multiple independently transmitted fragmented data units, and allocate corresponding multistream transmission channels to the multiple independently transmitted fragmented data units through sliding window control to obtain fragmented packets to be sent with transmission channel identifiers; Extract the header information and payload data of the fragmented packets to be sent, perform polynomial division on the header information and payload data, and use the remainder in the result as the check code; For a fragment packet to be sent with a channel identifier, a cyclic redundancy check algorithm is used to calculate the checksum of the fragment packet to be sent, and an asynchronous stream data block carrying the checksum is obtained. The asynchronous stream data block carrying the verification code is sent to the query processing terminal through an asynchronous transmission mechanism that supports multiplexing; Receive the receive status bits for the asynchronous stream data block returned by the query processing terminal; Determine if the receiving status bit is in the receiving completed state. If the receiving status bit is in the receiving completed state, obtain the integrity identifier generated after comparing the checksum extracted from the asynchronous stream data block by the query processing terminal. The integrity identifier is used to determine whether the asynchronous stream data block has passed the integrity check. If the integrity identifier indicates that the integrity check has passed, a data reception acknowledgment code for the corresponding asynchronous stream data block is generated and used as the data reception acknowledgment identifier. Obtain the data reception confirmation code, update the sliding position of the sliding window control based on the data reception confirmation code, and determine the next fragment packet to be sent.

9. A system for secure processing and transmission of massive amounts of data based on business processes, characterized in that, The system includes: The original business data extraction module is used to obtain the functional module code corresponding to the business process, perform static code parsing on the functional module code to obtain data query range configuration information containing data source location identifier and query condition parameters; according to the data query range configuration information, it obtains massive data records related to the business process from the distributed data storage system, performs field-level parsing on the massive data records, and extracts original business data containing user identification information and transaction flow information. The sensitivity feature assessment module is used to assess the sensitivity features of the data fields of the original business data using a machine learning classification model, and to classify the data fields into sensitive business data and non-sensitive business data according to the information type they carry. The de-identified business data generation module is used to determine whether the field attributes of the sensitive business data belong to identity recognition information. If the field attributes of the sensitive business data belong to identity recognition information, an irreversible hash algorithm is used to perform one-way transformation processing on the sensitive business data to obtain de-identified business data. The confidential business data generation module is used to determine whether the field attributes of the sensitive business data belong to financial asset information. If the field attributes of the sensitive business data belong to financial asset information, a reversible cryptographic encryption algorithm is used to perform ciphertext conversion processing on the sensitive business data to obtain confidential business data. The target query response dataset construction module is used to construct the target query response dataset by sequentially concatenating anonymized business data, confidential business data, and non-sensitive business data according to the data structure template predefined by the business system. The data fragment generation module is used to obtain the data size of the target query response dataset, determine whether the data size exceeds the single transmission capacity limit, and if the data size exceeds the single transmission capacity limit, then the target query response dataset is divided into blocks and sliced ​​to obtain multiple independently transmitted data fragments. The data reception confirmation identifier receiving module is used to send multiple independently transmitted fragmented data units to the query processing terminal using an asynchronous transmission mechanism that supports multiplexing, and to receive the data reception confirmation identifier returned by the query processing terminal after performing integrity verification on the multiple independently transmitted fragmented data units.