Non-intrusive stock data encryption method and device, equipment and storage medium

Abstract

The invention discloses a non-intrusive stock data encryption method and device, equipment and a storage medium, and relates to the field of data processing. Performing static analysis on a program in the static analysis engine through a reflection scanning technology, and determining a target field for processing stock data in the database; periodically querying and extracting stock data from the database, and carrying out slicing processing; performing plaintext judgment on the slice data by utilizing a dynamic detection method and a judgment rule; the dynamic detection method is determined based on a coding mode of preorder encryption; and performing transactional encryption processing on the recognized plaintext data according to the target encryption algorithm and the matched target field, and then storing the data back to the database. Target fields are automatically recognized through the reflection scanning technology, accurate plaintext judgment is achieved in combination with dynamic detection and a rule engine, data consistency is guaranteed through transactional encryption processing, and the method has the advantages that stock plaintext data are automatically recognized and encrypted, the system transformation risk is avoided, and the plaintext judgment accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a non-invasive existing data encryption method, device, equipment and storage medium. Background Art

[0002] Desensitization technology is a method of masking data through annotations. Data masking is a data security technology designed to transform sensitive information in raw data using pre-defined rules and algorithms to protect personal privacy and commercial secrets. Annotation desensitization technology automatically processes sensitive information by specifying masking rules through annotations in the code. Its core concept is to separate masking rules from business logic, improving code maintainability and scalability.

[0003] In current banking, finance, healthcare, and government data businesses, especially those that need to meet compliance requirements such as the Data Security Law, existing data processing scenarios in business systems require annotation desensitization technology to address the following issues when processing encrypted existing data: 1. Non-invasive encryption of existing data has its blind spots: Traditional annotation methods are only effective for newly generated data and cannot automatically identify and encrypt existing plaintext data in the database. This is because annotations can only mark fields that require encryption at the code level and are not directly effective for historical data already stored in the database. 2. System Intrusion Risk: Transformation requires modifying business code, which can lead to security risks such as service downtime and data inconsistency. This is because annotations must be added to existing code, which can cause system compatibility issues, such as conflicts between the MyBatis and Hibernate frameworks. 3. Inaccurate Plaintext Determination: Relying on a single annotation match can easily lead to misjudgments, failing to distinguish between encrypted and encoded data and actual plaintext. This is because annotations alone cannot accurately determine the actual state of the data and may misidentify encoded data as plaintext, resulting in duplicate encryption or decryption errors. Summary of the Invention

[0004] The present application provides a non-intrusive existing data encryption method, device, equipment and storage medium, which realize multimodal plaintext recognition and business-unaware encryption without blocking normal business requests.

[0005] In one aspect, the present application provides a non-intrusive method for encrypting stored data, the method comprising: Performing static analysis on the program in the static analysis engine using reflection scanning technology to determine target fields for processing the stock data in the database; the static analysis engine is constructed based on the data type of the database; Based on the target field, periodically query and extract stock data from the database and perform slicing processing; The slice data is plaintext judged using a dynamic detection method and a judgment rule; the dynamic detection method is determined based on the encoding method of the pre-order encryption, and the judgment rule is used to match different encrypted fields; The identified plaintext data is encrypted transactionally according to the target encryption algorithm and the matching target field and then stored back in the database.

[0006] Specifically, the static analysis engine is provided with a static model, and determines the target field by executing at least three layers of scanning procedures; the scanning process includes: Narrow the scan scope in the database by entity class package name; Use the set entity class table name @TableName annotation to determine the entity class corresponding to the database; Use the @Encrypt annotation of the set entity class field name to filter out the fields that need to be encrypted; The static model performs data query and extraction operations based on the reflection scanning technology determined by the three-layer scanning program.

[0007] Specifically, the periodic query and extraction of stock data from the database and slicing processing include: Extract N pieces of existing data from the database and store them in the buffer queue; Extract the switching data from the buffer queue in sequence and perform slicing processing; When all the slice data in the buffer queue are detected, N pieces of stock data are extracted from the database in a loop and the detection step is performed.

[0008] Specifically, the database includes several relational databases and unstructured databases; a data abstraction layer is also provided in the database, which has a built-in translation dictionary that matches all types of databases; In the stock data extraction stage, the translation dictionary translates the stock data of the database into slice data in the target format based on the basic sentences; In the encryption and storage phase, the translation dictionary translates the encrypted data back to the original format based on the basic sentence and stores it in the target type database.

[0009] Specifically, the use of dynamic detection methods and judgment rules to perform plaintext judgment on the slice data includes: Determine the encoding format corresponding to the pre-order encryption of the stock data, and use the encoding format to solve the slice data; Match the solution result with the preset encoding format. If the matching result is inconsistent, indicate that the current slice data is plain text; if the matching result is consistent, perform a secondary judgment based on the judgment rule; The judgment rules are built into the rule engine. When the rule engine is enabled, the target judgment rules are determined according to the source scenario of the stock data. Different judgment rules are used to process data from different source scenarios. The slice data is regularly matched based on the regular expression of the target judgment rule, and the data type is determined according to the matching result; wherein different data types correspond to different regular expressions, and when at least one set of regular expressions is successfully matched, the data type of the slice data is determined, and it is plaintext, otherwise the slice data is ciphertext.

[0010] Specifically, when it is determined that the slice data is plain text, it is transferred to a cache queue; the plain text data in the cache queue is encrypted according to a preset period; and the encrypted cipher text data is stored back in the database.

[0011] Specifically, when triggering database query and stock data extraction operations, a version ID is assigned to the data of the current transaction operation based on the multi-version concurrency control strategy; different version IDs of the same database form a version chain through pointers; different version IDs correspond to different transaction operations; When extracting existing data, the database does not lock all transaction operations; when performing encryption operations on the extracted existing data, the existing data in the database does not change; when writing back encrypted data, it does not lock, and does not change or block the existing data of other transaction operations.

[0012] Specifically, Introducing the @Encrypt annotation of the composite entity class data field into the static analysis engine and extracting the list of sensitive information types defined in the annotation; When performing plaintext judgment on the slice data according to the judgment rule, extracting the sensitive information type defined in the annotation based on the composite entity class data field annotation; Natural language processing technology is used to perform semantic segmentation on composite entity data fields, and a named entity recognition algorithm is used to identify the target fields corresponding to different types of sensitive information in the fields. Subsequently, the sliced ​​data is plaintext judged based on the target fields corresponding to different types of sensitive information. Specifically, the sliced ​​data is first solved. When the solution result cannot be matched, the target regularization rule is directly selected according to the sensitive information type to perform regularization matching.

[0013] In another aspect, the present application provides a non-intrusive stored data encryption device, the device comprising: A reflection scanning module is used to perform static analysis on the program in the static analysis engine using reflection scanning technology to determine the target fields for processing the stock data in the database; the static analysis engine is constructed based on the data type of the database; A slice extraction module is used to periodically query and extract stock data from the database based on the target field and perform slice processing; A ciphertext identification module, configured to perform plaintext identification on the slice data using a dynamic detection method and a judgment rule; the dynamic detection method is determined based on the encoding method of the pre-order encryption, and the judgment rule is used to match different encrypted fields; The encryption and storage module is used to perform transactional encryption processing on the identified plaintext data according to the target encryption algorithm and the matching target field and then store it back into the database.

[0014] On the other hand, the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the non-invasive inventory data encryption method described in the above aspect.

[0015] On the other hand, the present application provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the non-invasive inventory data encryption method described in the above aspects.

[0016] The beneficial effects brought about by the technical solution provided in the embodiment of the present application include at least: automatic identification of target fields through reflection scanning technology, accurate plaintext judgment achieved by combining dynamic detection and rule engine, and transactional encryption processing to ensure data consistency. It has the advantages of automatically identifying and encrypting existing plaintext data, avoiding system modification risks, and improving the accuracy of plaintext judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of traditional new data encryption provided by an embodiment of the present application; Figure 2 This is a flow chart of a non-invasive stock data encryption method provided by an embodiment of the present application; Figure 3 It is a schematic diagram of the data abstraction layer structure established for various database types; Figure 4 A detailed flow chart of the non-intrusive stock data encryption method is given; Figure 5The following is a structural block diagram of a non-intrusive stock data encryption device provided in an embodiment of the present application; Figure 6 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0019] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0020] This application is based on the traditional data encryption principle, which uses Java annotations (such as @Encrypt) to mark sensitive fields and uses AOP or ORM framework to automatically encrypt newly written data when persisting data. Figure 1 As shown, the typical process includes: 1. Development stage: Manually mark fields that need to be encrypted (such as @Encrypt marking data name and mobile phone number data); 2. Runtime: The intercepted data is matched to determine whether it contains class name information that matches the user entity class (annotated with @Encrypt) and is extracted; 3. Encryption: Perform symmetric encryption such as AES on the marked fields and then store them in the database; 4. Decryption: Decrypt in reverse order using the same key.

[0021] The common technical defects of this solution are mainly manifested as follows: 1. Existing data coverage blind spot: This feature is only effective for new data and cannot automatically process existing plaintext data. Manual migration scripts must be written.

[0022] 2. Highly invasive: Business code needs to be modified, which may cause system compatibility issues (such as MyBatis / Hibernate framework conflicts).

[0023] 3. Inaccurate plaintext determination: Relying on annotations, it is easy to misjudge encoded data (such as Base64) as plaintext, resulting in repeated encryption or direct decryption of existing plaintext data, causing errors.

[0024] The root cause of these problems is that traditional methods are passive encryption based on annotation desensitization, lack the ability to actively identify and process existing data, and also lack an intelligent plaintext judgment mechanism.

[0025] For example, in a core banking transaction system built on the SpringBoot framework, when the customer ID number field is marked with the @Encrypt annotation, only newly entered data can be encrypted, while the 2 million historical customer information items already stored in the database remain in plain text. When the system connects to an unstructured database to store electronic contracts, the traditional annotation method cannot identify sensitive information embedded in PDF documents due to the lack of a unified field mapping mechanism. During the data extraction phase, the serial number field encoded in Base64 was mistakenly identified as encrypted data because its encoding characteristics are similar to those of encrypted data, resulting in the actual plaintext data not being processed. During the system upgrade process, service interruptions occurred due to conflicts in MyBatis entity class annotations, and compatibility issues occurred between the data encryption module and the ORM framework.

[0026] When faced with the above problems, this application first considers how to establish a stock data identification mechanism without modifying the existing business code. If the traditional solution directly modifies the database structure, it will cause system intrusion risks, and it is difficult to accurately locate the target field by relying solely on dynamic scanning. To this end, this application attempts to combine reflection scanning technology with a static analysis engine, and maps database fields by reverse parsing the program entity class structure, where the database adaptation characteristics of the static analysis engine are compatible with heterogeneous data sources. In order to solve the system load problem that may be caused by periodic data processing, the collaborative mechanism of batch slicing extraction and buffer queues was explored, and dynamic resource allocation was achieved through transactional operations. In response to the problem of plaintext misjudgment, it was found that the pre-order encryption encoding format is traceable. Using it as a dynamic detection benchmark can inherit historical encryption strategies, and at the same time introduce a multi-rule regular matching mechanism to deal with complex data types. The secondary judgment logic of the rule engine can effectively distinguish between encoded data and real plaintext.

[0027] To solve the above problems, this application proposes a method for encrypting stock data based on dual-engine drive, such as Figure 1 As shown, it includes the following steps: S1. Perform static analysis on the program in the static analysis engine using reflection scanning technology to determine the target fields for processing the existing data in the database; Reflection scanning technology refers to the technology of dynamically parsing the program code structure through the reflection mechanism to identify the target field. Specifically, it can be implemented by scanning the annotation information of the entity class fields using the Java reflection API. This technology can obtain the class structure and field properties without modifying the source code.

[0028] A static analysis engine is a parsing tool that automatically builds a data model based on the database schema. This is achieved by analyzing SQL statement structures using an abstract syntax tree (AST). The engine ensures field recognition compatibility by adapting to different database types. The static analysis engine in this solution is built based on database data types, using a hierarchical structure specifically tailored to the database data type to establish reflective scanning relationships.

[0029] For example, in a banking data processing business, the target field may be information such as the customer's mobile phone number and / or name.

[0030] S2. Based on the target field, periodically query and extract existing data from the database and perform slicing processing; Periodic query extraction refers to the process of retrieving database records in batches at set intervals. This can be achieved using a scheduled task scheduling framework coupled with paging query SQL statements. This mechanism reduces memory usage by controlling the amount of data processed at a time. Slicing processing is the process of dividing batches of data into independent processing units. This can be achieved by using a memory partitioning algorithm to divide the dataset into fixed-size data segments. This operation facilitates parallel processing and exception recovery. Because the amount of data that meets the requirements in the database is quite large, slicing can prevent excessive processing load caused by excessive data. For example, only 1,000 records can be retrieved at a time and the processing can be looped until completion.

[0031] This rolling sharding mechanism ensures the stability and efficiency of large-scale data processing. By controlling the amount of data processed each time, it avoids excessive use of system resources while ensuring the interruptibility and recoverability of the processing process.

[0032] S3. Use dynamic detection methods and judgment rules to perform plaintext judgment on slice data; the dynamic detection method is determined based on the encoding method of the pre-order encryption, and the judgment rules are used to match different encrypted fields; Dynamic detection methods refer to algorithms that determine data status based on historical encryption characteristics. This can be achieved by using codecs like Base64 and AES to verify the data format. This method ensures the continuity of the judgment logic by inheriting the previous encryption method. Judgment rules refer to a set of matching conditions for multi-dimensional identification of data types. This can be achieved by using a regular expression rule library in conjunction with a rule engine to perform pattern matching. Different encrypted fields are identified by the same matching relationship and then matched. This rule system improves the accuracy of plaintext recognition through multi-factor verification.

[0033] S4. The identified plaintext data is encrypted transactionally according to the target encryption algorithm and the matching target field and then stored back in the database.

[0034] Once the plaintext data has been identified, it needs to be encrypted. This encryption method can use the same algorithm as the previous one to maintain the encryption consistency of the existing data. Of course, different encryption algorithms can also be used depending on the data type or importance. The encrypted data can then be stored back in the database.

[0035] Transactional encryption processing refers to an execution mechanism that ensures the atomicity of data encryption operations. Specifically, it can be implemented by using a database transaction management module to control the encryption write-back process. This processing method avoids data lock contention through version chain control.

[0036] In summary, this application, by building a dual mechanism of reflection scanning and dynamic detection, automatically identifies target fields through a static analysis engine to eliminate code intrusion, combines the detection of inheritance of previous encryption encoding formats with a multi-rule matching strategy, and achieves high-precision identification of plaintext judgment during the encryption of stock data. At the same time, transactional version control is used to ensure data consistency between encryption operations and business systems. This non-invasive architecture achieves accurate encryption of stock data, establishes a dual mechanism of static analysis and dynamic detection, and effectively solves the problems of blind spots in non-invasive stock data encryption, system intrusion risks, and inaccurate plaintext judgment.

[0037] As a preferred embodiment, the solution of this application is specifically implemented as follows: First, we built a static analysis engine. This engine was designed based on the target database's data type and included adapter modules for both relational and unstructured databases. The engine also included a built-in reflection scanner capable of parsing class structures in various programming languages, including Java and Python.

[0038] Next, a reflection scan is performed. Starting from the project root, the scanner traverses all class files layer by layer. For each class file, information such as the package name, class name, and field names is extracted. Particular attention is paid to classes and fields annotated with annotations such as @TableName and @Column, which typically correspond to database tables and columns.

[0039] Next, determine the target fields. Compare the scan results with the database schema to establish a mapping between entity class fields and database columns. For projects that don't use an ORM framework, analyze the table and column names in the SQL statements to infer the target fields.

[0040] Due to the complex database structure and numerous entity classes, static analysis engines may face problems such as excessive scanning scope, unclear correspondence between entity classes and database tables, and inefficient screening of encrypted fields, resulting in inaccurate target field identification or excessive resource consumption. To address this, the present application further proposes that a static model be provided in the static analysis engine to determine the target field by executing at least three layers of scanning procedures; the scanning process includes: 1. Narrow the scan scope in the database by the entity class package name; 2. Use the set entity class table name @TableName annotation to determine the entity class corresponding to the database; 3. Use the @Encrypt annotation of the set entity class field name to filter out the fields that need to be encrypted; During formal execution, the static model performs data query and extraction operations based on the reflective scanning technique determined by the three-tier scanning procedure.

[0041] As a preferred embodiment, the solution of this application is specifically implemented as follows: The static analysis engine sets up a static model and determines the target fields by executing a three-tiered scanning process. The first tier narrows the database scan scope by entity class package names, for example, specifying all classes under the com.example.entity package. The second tier uses the @TableName annotation to identify the entity class corresponding to the database. For example, @TableName("user_info") annotates the User class to correspond to the user_info table. The third tier uses the @Encrypt annotation to select fields that require encryption. For example, annotating the name field with @Encrypt indicates that this field requires encryption. The static model performs data query and extraction operations based on the reflection scanning technology determined by these three tiers of scanning.

[0042] In practice, the Java Reflection API can be used to obtain class metadata. First, all classes in the specified package are loaded using the ClassLoader. Then, these classes are iterated over to check for the presence of the @TableName annotation. For classes annotated with @TableName, the fields are further iterated over to identify those annotated with @Encrypt. Finally, this information is stored in a static model for subsequent data operations.

[0043] For example, for the User class: @TableName("user_info")public class User {@Encrypt private Stringname;private int age;}; The static model records the information that needs to be encrypted in the name field of the user_info table. When executing data queries, the static model can dynamically construct SQL statements based on this information, selecting only the fields that need to be processed, thereby improving query efficiency.

[0044] Through the above technical solution, this application achieves accurate and efficient target field identification. Through a three-layer scanning program, the scanning range is gradually narrowed, and the efficiency of static analysis is improved. The annotation mechanism is used to clarify the correspondence between entity classes and database tables, solving the problem of unclear correspondence between entity classes and database tables. At the same time, the use of the @Encrypt annotation simplifies the screening process of encrypted fields and improves the efficiency of encrypted field screening. This method not only reduces the resource consumption of the static analysis engine, but also improves the accuracy of target field identification, laying the foundation for subsequent encryption processing of stock data.

[0045] As mentioned above, the amount of data filtered from the database is massive. Extracting all of the existing data at once may lead to system resource exhaustion or processing delays. The integrity and continuity of data processing cannot be guaranteed, and the excessive amount of data may lead to memory overflow or transaction conflicts. In response to this, the present application further proposes a technical solution that extracts N pieces of existing data from the database and stores them in a buffer queue. Switching data is sequentially extracted from the buffer queue for slicing processing. After all sliced ​​data in the buffer queue has been inspected, the next batch of N pieces of existing data is cyclically extracted and the inspection step is performed.

[0046] Among them, the amount of data N extracted at a time can be set to a range of 100 to 1000, and the specific value is dynamically adjusted according to the maximum concurrency of the database connection pool and the server memory capacity. The buffer queue is implemented using a first-in-first-out data structure, and the queue capacity can be configured to 1.5 to 2 times N to accommodate abnormal retry data. A flow control valve is set between the slice processing unit and the buffer queue, and the data extraction operation is suspended when the queue filling rate reaches 80%. The cyclic extraction mechanism is implemented by setting a breakpoint resume mark, and the unique identifier of the last data is recorded each time it is extracted as the starting condition for the next query. The rolling loop sharding processing method can well balance the contradiction between system resources and execution efficiency.

[0047] Some of the aforementioned solutions in this application propose encrypting existing data through reflection scanning and slicing. However, in scenarios involving multiple database types, due to differences in data formats across different databases, direct extraction and restoration of encrypted data can lead to format incompatibility, resulting in data translation errors or inefficient processing. For example, relational databases like MySQL and Oracle, and unstructured databases like MongoDB, differ in their data storage and processing methods, making direct manipulation impossible.

[0048] In this regard, this application proposes a data abstraction layer structure for various database types, such as Figure 3As shown, the data abstraction layer structure has a built-in translation dictionary that matches all types of databases. The data abstraction layer is configured as a unified interface layer across database types, and its built-in translation dictionary contains multiple sets of predefined syntax conversion rules. The basic statements of the translation dictionary can be extended based on international standards, such as using the SQL-92 standard as the conversion benchmark for relational databases, while integrating BSON format parsing rules to adapt to unstructured databases. During the data extraction process, the translation dictionary parses the query statement structure of the original database and converts query results of different syntax forms into an intermediate slice format. The intermediate slice format is designed to be a standardized structure containing field names, data types, and binary values. During the encrypted storage phase, the translation dictionary reversely parses the intermediate slice format according to the target database type and generates insert statements that comply with the target database syntax specifications, such as converting binary encrypted data into MongoDB's BinData type or MySQL's BLOB type.

[0049] Specifically, during the stock data extraction phase, the translation dictionary first identifies the type of the source database and loads the corresponding syntax parsing module. For example, when the source database is Oracle, the parsing module will identify its unique TO_DATE function format and convert it into a timestamp type in the intermediate slice format; when the source database is MongoDB, the parsing module will process the nested document structure and break it down into slice data for multiple independent fields. During the encrypted restoration phase, the translation dictionary selects the corresponding syntax generation module based on the target database type, for example, reassembling the encrypted slice data into PostgreSQL's JSONB type or Access's Memo field format. Through a bidirectional conversion mechanism for basic statements, the data abstraction layer shields the differences between different databases in data type definitions, query syntax, and storage structures, eliminating the need for database-specific adaptation modifications to the encryption process. As a result, in stock data processing scenarios involving mixed database environments, encryption failures due to format conversion errors can be avoided while reducing the syntax compatibility verification overhead during cross-database operations, thereby improving the processing efficiency and applicability of the encryption system.

[0050] In some embodiments, using the pre-order encryption coding format to solve the match is prone to misjudgment, and it is impossible to effectively distinguish between real plaintext and encoded data or encrypted data. When the solution result cannot match, there is a lack of a secondary judgment mechanism, and there is a technical defect of insufficient plaintext judgment accuracy.

[0051] In this regard, the present application further proposes a method for performing plaintext judgment on slice data using a dynamic detection method and judgment rules, comprising the following steps: 1) Determine the encoding format corresponding to the pre-order encryption of the existing data, and use the encoding format to solve the slice data; The encoding format can be solved using a hexadecimal decoder or a Base64 decoder. If the preceding encryption is performed using the hexadecimal encoding format HexUtil.encodeHex(), the HexUtil.isHexNumber() method is called first to verify the format and determine whether it is in hexadecimal.

[0052] 2) Match the solution result with the preset encoding format; If the matching results are inconsistent, the current slice data is plaintext; if the matching results are consistent, the current slice data may be ciphertext or plaintext. For example, if the data is uniformly encoded using HexUtil.encodeHex(), the result must be hexadecimal. If the result is not hexadecimal, it is consistent with the original encoding format, and encryption is performed. However, if the matching data is in hexadecimal, it does not necessarily mean that the original data is encrypted, so a secondary judgment based on the rule engine is required.

[0053] 3) Perform regular matching on the slice data based on the regular expression of the target judgment rule, and determine the data type according to the matching result.

[0054] There are many types of judgment rules, primarily built into the rules engine. When the rules engine is enabled, the target judgment rules are determined based on the source scenario of the existing data, and different judgment rules are used to process data from different source scenarios. Furthermore, the judgment rules built into the rules engine must form a data format alignment mechanism with the translation dictionary of the data abstraction layer.

[0055] For example, in banking scenarios, the standardized data output by the translation dictionary can be directly matched using the bank card number regular expression ^[0-9]{16,19}$. The regular expression library includes three types of expressions: numeric, character, and mixed. Numeric expressions can be set to ^\d+$, character expressions to ^[\u4e00-\u9fa5]+$, and mixed expressions to ^[A-Za-z0-9]+$. If the solution fails, the rule engine automatically calls the regular expression group corresponding to the data abstraction layer output format.

[0056] Figure 4 A detailed flowchart of a non-invasive method for encrypting existing data is provided. The data of interest is primarily customer names, bank card numbers, and mobile phone numbers. Since customer names are character-based data, and mobile phone and bank card numbers are numeric, this banking scenario requires both character-based and numeric-based regular expressions.

[0057] For slice data that fails the initial match, it is not yet known which regularization expression is applicable, so this step uses a one-to-one matching method. For the target slice data, when at least one set of regularization expressions successfully matches, the data type of the slice data is determined accordingly, and the target slice data is determined to be plaintext. Otherwise, the slice data is ciphertext.

[0058] Specifically, after the data abstraction layer completes format standardization, it first attempts to reverse-engineer the sliced ​​data using the pre-order encryption encoding format. If the result does not match the pre-set encoding format, it is directly judged as plaintext data. If the decoding fails completely, the rule engine activates the corresponding regular expression group based on the data type marked by the data abstraction layer.

[0059] For example, in banking scenarios, hexadecimal data such as mobile phone numbers and bank card numbers are matched against the regular expressions ( / ^(13[0-9]|14[01456879]|15[0-35-9]|16

[2567] |17[0-8]|18[0-9]|19[0-35-9])\d{8}$ / and / ^([1-9]{1})(\d{15}|\d{16}|\d{18})$ / ). If either of these matches, the data is considered plaintext. This dual-check mechanism, combined with format standardization at the data abstraction layer, increases the accuracy of regular expression matching to over 98% and reduces the false positive rate to below 0.5%. Furthermore, the rule engine's dynamic loading mechanism supports online updates of the regular expression library, enabling real-time adaptation to the recognition needs of new data formats.

[0060] The same is true for string types. If the regularized matching result is Chinese, it means that the stored data is plain text and the encryption process should be used. If the matching result is not Chinese, it means that it is not ciphertext or other unrecognizable data and can be ignored.

[0061] In some embodiments, directly encrypting and storing each plaintext data item individually during periodic processing can lead to a surge in database write pressure. In concurrent scenarios, conflicts between encrypted transactions and business operations can lead to inconsistent data versions. To address this, the present application further proposes transferring slice data to a cache queue when it is determined to be plaintext, encrypting the plaintext data in the cache queue according to a preset period, and storing the encrypted ciphertext data back in the database.

[0062] The cache queue uses a first-in, first-out data structure, and the queue capacity can be set to dynamically adjust, for example, automatically expanding to 1,000-5,000 records based on the current database load. The preset period trigger mechanism can be configured to a fixed time window, such as performing a batch encryption operation every 5 minutes.

[0063] Specifically, after slicing and determination, plaintext data is temporarily stored in a cache queue. When the queue capacity threshold is reached, data inflow is automatically suspended to prevent overflow. When the preset period expires, the encryption module extracts data from the head of the queue in batches, for example, 200 entries at a time, for parallel encryption. The encryption process uses an algorithm that matches the target field, such as the SM4 algorithm for the ID card field and the AES algorithm for the phone number field.

[0064] In some of the solutions mentioned above, traditional database transaction processing mechanisms can block other concurrent operations when locking existing data, resulting in reduced system throughput. Furthermore, directly modifying the original data during the encrypted restore phase can cause version conflicts and data consistency risks. To address this, this application implements a multi-version concurrency control strategy based on the MVCC multi-version concurrency control mechanism.

[0065] When triggering database queries and extracting existing data, a version ID is assigned to the data being operated on based on a multi-version concurrency control strategy. Different version IDs within the same database form a version chain through pointers, and different version IDs correspond to different transaction operations. When extracting existing data, the database does not lock all transaction operations. When encrypting the extracted existing data, the existing data in the database remains unchanged. When writing back encrypted data, no locks are applied, preventing the existing data from being altered or blocked by other transaction operations.

[0066] When triggering a database query or extracting existing data, a version ID is assigned to the data in the current transaction based on a multi-version concurrency control strategy. For example, a timestamp or an incrementing integer can be used as the version ID. Different version IDs within the same database form a version chain through pointers, forming a linked list structure. Different version IDs correspond to different transaction operations, and each transaction operation creates a new version.

[0067] When extracting existing data, the database does not lock any transactions. Specifically, an optimistic locking mechanism can be used, allowing multiple transactions to read data simultaneously while performing version checks on writes. When encryption is performed on the extracted existing data, the existing data in the database remains unchanged. The encryption operation is performed in memory and does not directly modify the original data.

[0068] When writing back encrypted data, no locks are applied, and existing data is not altered or blocked by other transactions. This can be achieved by creating a new data version, writing the encrypted data to the new version, and updating the version chain. Other transactions can still access the old version of the data until they are completed or committed.

[0069] Furthermore, a garbage collection mechanism can be set up to regularly clean up old versions of data that are no longer referenced by any transactions to optimize storage space.

[0070] Through the above technical solution, the present application realizes efficient non-intrusive encryption of existing data. In this way, the performance bottleneck caused by the traditional locking mechanism is avoided and the concurrent processing capability of the system is improved. At the same time, business continuity is guaranteed, encryption tasks can be evenly distributed, and no excessive pressure is placed on a single node. In addition, the impact on the business is minimized. During the encryption process, normal business requests will not be blocked, and true business-unaware encryption is achieved. Specifically, through the version isolation mechanism, direct locking of the original data is avoided, and the availability of the database is guaranteed. The encryption operation only acts on independent versions of the data copy to ensure that the original data is not tampered with. In the write-back phase, the new version of the ciphertext data is linked through the version chain pointer, while maintaining lock-free writing to prevent other transactions from blocking access to historical version data due to encryption and storage operations. This method not only meets the encryption processing requirements, but also realizes concurrent isolation and data consistency between transactions.

[0071] In some special usage scenarios, when the data fields in the generated stock data are composite structures containing multiple sensitive information, it is impossible to effectively separate the different sensitive types in the mixed data based on a single field annotation, resulting in the inability to accurately identify the plaintext information nested in the composite field, causing semantically related sensitive data to be In this regard, the present application further proposes that a static model is set in the static analysis engine, the @Encrypt annotation of the composite entity class data field is introduced in the static analysis engine, and the list of sensitive information types defined in the annotation is extracted; When performing plaintext judgment on the slice data according to the judgment rule, extracting the sensitive information type defined in the annotation based on the composite entity class data field annotation; Natural language processing technology is used to perform semantic segmentation on composite entity data fields, and a named entity recognition algorithm is used to identify the target fields corresponding to different types of sensitive information in the fields. Subsequently, the sliced ​​data is plaintext judged based on the target fields corresponding to different types of sensitive information. Specifically, the sliced ​​data is first solved. When the solution result cannot be matched, the target regularization rule is directly selected according to the sensitive information type to perform regularization matching. Among them, the @Encrypt annotation of the composite entity class data field can be configured at the entity class field declaration. The annotation contains a sensitive information type list parameter, for example, marking the address field as containing sensitive information such as "name", "telephone", "street", and "bank card". The semantic segmentation stage adopts a sequence labeling model based on a bidirectional long short-term memory network to decompose the composite field content into independent segments according to semantic units. The segmentation granularity can be controlled at the character level or word level. The named entity recognition algorithm is built based on the conditional random field model. Its label set corresponds one-to-one with the sensitive type declared in the @Encrypt annotation. For example, the segmented "Zhang San" is identified as a name type entity. The regularization rule library stores regular expressions corresponding to different sensitive types. For example, the telephone number type corresponds to a regular pattern containing an area code and a separator. The matching threshold can be set to above 85%.

[0072] Because this solution can directly map different types of sensitive data to corresponding regular expressions, there is no one-to-many matching in the secondary determination stage. For example, if @Encrypt_phonexx is defined as a phone number (target field) type data and @Encrypt_namexx is defined as a name (target field) type data based on sensitive fields, then directly matching with numeric and character regular expressions is sufficient. This can significantly improve matching efficiency.

[0073] Through the above technical solutions, this application effectively solves the technical defect of incomplete recognition of nested sensitive information in composite fields. By guiding the direction of semantic segmentation through structured annotations, the problem of missed recognition of sensitive units caused by blurred semantic boundaries in traditional methods is avoided. The regular rule switching mechanism based on type association eliminates the risk of misjudgment of specific sensitive types in composite data by general matching rules. Through differentiated encryption strategies, accurate isolation and processing of sensitive and non-sensitive data within a field is achieved, while ensuring data security and maintaining the business availability of non-sensitive information. The adoption of structured storage formats provides clear semantic tags for subsequent data analysis, reducing the parsing complexity when using data.

[0074] This embodiment solution is applied to the existing data encryption transformation project of the smart union. The project involves an employee file database containing 250 million sensitive data items (such as ID numbers, mobile phone numbers, and physical examination records), which must meet the Level 3 security protection requirements of the "Data Security Law".

[0075] In this case, the server was configured with an 8-core CPU and 16GB of memory. The static analysis engine scanned the path com.gov.zhgh.domain, which contained 128 entity classes. It identified @Encrypt-annotated fields in 42 tables, totaling 218 sensitive fields (such as id_card and phone), in a time-consuming 0.25 hour (traditional manual labeling would require 3 man-days).

[0076] Dynamic detection and the rules engine initially identified 20 million pieces of data. Then, using a pre-built regular expression library based on multimodal judgments, including ID card numbers (^\d{17}[\dXx]$) and mobile phone numbers (^1[3-9]\d{9}$), the algorithm identified 170 million pieces of data. The total processing time was three hours.

[0077] Peak memory usage for encrypted storage: 1.2 GB, cumulative time consumption: 5.75 hours.

[0078] The traditional manual method takes 312 hours (13 days), while the method of the present invention takes only 9 hours.

[0079] Through experiments and rigorous comparative testing, this solution has proven to solve the problem of plaintext encryption for existing data. It achieves 99.5% accuracy and supports re-execution, preventing interruptions and subsequent inability to continue. Furthermore, the execution process does not impact existing business operations. In actual testing of core financial systems, the encryption process did not require any business code modifications, resulting in zero system restarts, completely eliminating the risk of service interruptions caused by code changes.

[0080] In summary, the implementation of this solution can bring the following technical effects: Automated encryption of existing data: Establish a dynamic mapping between code features and data entities to encrypt existing data without modifying business code.

[0081] Business-unaware encryption: Based on MVCC multi-version control and distributed task scheduling, the encryption process ensures that the table is not locked and normal business requests are not blocked.

[0082] Compatibility with heterogeneous environments: Adapts to different database types through a unified abstraction layer, supporting unified cross-platform management of encryption policies.

[0083] Figure 5 The following is a structural block diagram of a non-intrusive stock data encryption device provided in an embodiment of the present application, wherein the device includes: Reflection scanning module 510, used to perform static analysis on the program in the static analysis engine using reflection scanning technology to determine the target fields for processing the stored data in the database; the static analysis engine is constructed based on the data type of the database; Slice extraction module 520, for periodically querying and extracting stock data from the database based on the target field, and performing slicing processing; The ciphertext identification module 530 is used to perform plaintext identification on the slice data using a dynamic detection method and a judgment rule; the dynamic detection method is determined based on the encoding method of the pre-order encryption, and the judgment rule is used to match different encrypted fields; The encryption and storage module 540 is used to perform transactional encryption processing on the identified plaintext data according to the target encryption algorithm and the matching target field and then store the data back into the database.

[0084] The non-invasive existing data encryption device provided in the embodiment of the present application can be applied to the non-invasive existing data encryption method provided in the above embodiment. For relevant details, please refer to the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0085] It should be noted that the non-invasive stock data encryption device provided in the embodiments of the present application is merely illustrated by the division of the above-mentioned functional modules / functional units. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the non-invasive stock data encryption device can be divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the non-invasive stock data encryption method provided in the above-mentioned method embodiment and the implementation method of the non-invasive stock data encryption device provided in this embodiment are based on the same concept. The specific implementation process of the non-invasive stock data encryption device provided in this embodiment is detailed in the above-mentioned method embodiment and will not be repeated here.

[0086] Figure 6 The following is a block diagram of the structure of a computer device provided by an exemplary embodiment of the present application. The computer device is a desktop computer, a laptop computer, a PDA, a cloud server, and the like. The computer device may include, but is not limited to, a processor and a memory. The processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processing units (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0087] The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor may be implemented in at least one of the following hardware forms: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is used to process data while awake, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data while in standby mode. In some embodiments, the processor may integrate a graphics processing unit (GPU), which is responsible for rendering and drawing content displayed on the display. In some embodiments, the processor may also include an artificial intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0088] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, implementing the method in the above-mentioned method embodiment. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; and the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0089] In some embodiments, the computer device may optionally include a peripheral device interface and at least one peripheral device. The processor, memory, and peripheral device interface may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface via a bus, signal lines, or circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit, a display screen, and a keyboard.

[0090] The peripheral device interface can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor and memory. In some embodiments, the processor, memory, and peripheral device interface are integrated on the same chip or circuit board. In other embodiments, any one or two of the processor, memory, and peripheral device interface can be implemented on separate chips or circuit boards, although this embodiment is not limited to this.

[0091] The display screen is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. If the display screen is a touch screen, it also has the ability to capture touch signals on or above the display screen. These touch signals can be input as control signals to a processor for processing. In this case, the display screen can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard.

[0092] Those skilled in the art will understand that the structure shown in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0093] The embodiment of the present application also discloses a computer-readable storage medium. Specifically, the computer-readable storage medium is used to store a computer program, and when the computer program is executed by the processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art will understand that all or part of the processes in the above-mentioned method implementation of the present application can be completed by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the implementation of the above-mentioned methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0094] This specific embodiment is merely an explanation of the present invention and is not intended to limit the present invention. After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed. However, as long as such modifications are within the scope of the claims of the present invention, they are protected by patent law.

Claims

1. A non-invasive stock data encryption method, characterized in that: The method comprises: Performing static analysis on the program in the static analysis engine using reflection scanning technology to determine target fields for processing the stock data in the database; the static analysis engine is constructed based on the data type of the database; Based on the target field, periodically query and extract stock data from the database and perform slicing processing; The slice data is plaintext judged using a dynamic detection method and a judgment rule; the dynamic detection method is determined based on the encoding method of the pre-order encryption, and the judgment rule is used to match different encrypted fields; The identified plaintext data is encrypted transactionally according to the target encryption algorithm and the matching target field and then stored back in the database.

2. The method according to claim 1, characterized in that The static analysis engine is provided with a static model, and determines the target field by executing at least three layers of scanning procedures; the scanning process includes: Narrow the scan scope in the database by entity class package name; Use the set entity class table name @TableName annotation to determine the entity class corresponding to the database; Use the @Encrypt annotation of the set entity class field name to filter out the fields that need to be encrypted; The static model performs data query and extraction operations based on the reflection scanning technology determined by the three-layer scanning program.

3. The method according to claim 1, characterized in that The periodic query and extraction of stock data from the database and slicing processing include: Extract N pieces of existing data from the database and store them in the buffer queue; Extract the switching data from the buffer queue in sequence and perform slicing processing; When all the slice data in the buffer queue are detected, N pieces of stock data are extracted from the database in a loop and the detection step is performed.

4. The method according to claim 1, wherein The database includes several relational databases and unstructured databases; a data abstraction layer is also provided in the database, which has a built-in translation dictionary that matches all types of databases; In the stock data extraction stage, the translation dictionary translates the stock data of the database into slice data in the target format based on the basic sentences; In the encryption and storage phase, the translation dictionary translates the encrypted data back to the original format based on the basic sentence and stores it in the target type database.

5. The method according to claim 4, characterized in that The method of using a dynamic detection method and a judgment rule to perform plaintext judgment on the slice data includes: Determine the encoding format corresponding to the pre-order encryption of the stock data, and use the encoding format to solve the slice data; Match the solution result with the preset encoding format. If the matching result is inconsistent, indicate that the current slice data is plain text; if the matching result is consistent, perform a secondary judgment based on the judgment rule; The judgment rules are built into the rule engine. When the rule engine is enabled, the target judgment rules are determined according to the source scenario of the stock data. Different judgment rules are used to process data from different source scenarios. The slice data is regularly matched based on the regular expression of the target judgment rule, and the data type is determined according to the matching result; wherein different data types correspond to different regular expressions, and when at least one set of regular expressions is successfully matched, the data type of the slice data is determined, and it is plaintext, otherwise the slice data is ciphertext.

6. The method according to any one of claims 1 to 5, characterized in that: When it is determined that the slice data is plain text, it is transferred to a cache queue; the plain text data in the cache queue is encrypted according to a preset period; and the encrypted cipher text data is stored back in the database.

7. The method according to claim 6, characterized in that When triggering database query and stock data extraction operations, a version ID is assigned to the data of the current transaction based on the multi-version concurrency control strategy; different version IDs of the same database form a version chain through pointers; Different version IDs correspond to different transaction operations; When extracting existing data, the database does not lock all transaction operations; when performing encryption operations on the extracted existing data, the existing data in the database does not change; when writing back encrypted data, it does not lock, and does not change or block the existing data of other transaction operations.

8. A non-intrusive stock data encryption device, characterized in that: The device comprises: A reflection scanning module is used to perform static analysis on the program in the static analysis engine using reflection scanning technology to determine the target fields for processing the stock data in the database; the static analysis engine is constructed based on the data type of the database; A slice extraction module is used to periodically query and extract stock data from the database based on the target field and perform slice processing; A ciphertext identification module, configured to perform plaintext identification on the slice data using a dynamic detection method and a judgment rule; the dynamic detection method is determined based on the encoding method of the pre-order encryption, and the judgment rule is used to match different encrypted fields; The encryption and storage module is used to perform transactional encryption processing on the identified plaintext data according to the target encryption algorithm and the matching target field and then store it back into the database.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the non-invasive stock data encryption method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the non-invasive stock data encryption method as described in any one of claims 1 to 7.

Images