Data processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a program product, and relates to the technical field of big data processing. The unstructured data is written into a two-dimensional table, and the two-dimensional table is determined based on a preset data classification and grading list; analyzing an association relationship between the unstructured data and a preset data classification and grading list, and generating a mapping rule; and marking the unstructured data stored in the two-dimensional table according to a mapping rule and the preset data classification and grading list to obtain a marking result, the marking result comprising a data category and a data level corresponding to the unstructured data. According to the method, a two-dimensional table is constructed based on a preset classification and grading list, and records are marked in combination with a mapping rule, so that effective classification and grading of unstructured data are realized; and meanwhile, by adopting a mapping rule fusing keywords, regularization and semantic recognition, the classification and grading accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to data processing methods, apparatus, equipment, storage media and program products. Background Technology

[0002] For particularly important matters, which involve a large number of unstructured documents of different security levels, the security classification of these documents is usually high. If any oversight occurs, it may have an immeasurable impact on the interests of the enterprise and the security of government information.

[0003] Current data classification and grading methods primarily focus on structured data, emphasizing the organization and management of data with standardized formats and clearly defined fields. This bias makes unstructured documents easily marginalized during the classification and grading process, resulting in incomplete data coverage. More importantly, some core and important data may therefore remain unmanaged or overlooked, creating potential security risks and posing hidden dangers to government information security.

[0004] Therefore, existing technologies lack a method for accurately and effectively classifying and grading unstructured data to improve the security level of government information.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a data processing method, apparatus, device, storage medium, and program product, which aims to solve the technical problem of how to accurately and effectively classify and grade unstructured data.

[0007] To achieve the above objectives, this application proposes a data processing method, which includes: Acquire unstructured data; The unstructured data is written into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list; Analyze the relationship between the unstructured data and the preset data classification and grading list to generate mapping rules; According to the mapping rules and the preset data classification and grading list, the unstructured data stored in the two-dimensional table is tagged to obtain the tagging result, wherein the tagging result includes the data category and data level corresponding to the unstructured data.

[0008] In one embodiment, the step of writing the unstructured data into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list, includes: Based on the data categories, grading standards, and corresponding data characteristics defined in the preset data classification and grading list, determine the field information and corresponding field types of the two-dimensional table; A two-dimensional table is constructed based on the field information and field types; The unstructured data is cleaned to remove duplicate data and correct data with format errors, resulting in cleaned data. The cleaned data is written into the two-dimensional table using database data extraction, transformation, and loading ETL tools.

[0009] In one embodiment, after the step of writing the cleaned data into the two-dimensional table using a database data extraction, transformation, and loading ETL tool, the method further includes: The data in the two-dimensional table is sampled and checked to verify the matching degree between the data and the preset data classification and grading list, the integrity of the table structure, and the consistency of the data.

[0010] In one embodiment, the step of analyzing the association between the unstructured data and the preset data classification and grading list to generate mapping rules includes: For each data category and corresponding grading standard in the preset classification and grading list, identification rules for identifying the unstructured data are determined, wherein the identification rules include one or more of keyword rules, regular expression rules, or semantic recognition rules; The identification rules are associated with the corresponding data categories and grading standards to form the mapping rules.

[0011] In one embodiment, after the step of associating the identification rules with the corresponding data categories and grading standards to form the mapping rules, the method further includes: The identification rules and the mapping rules are stored in a pre-built rule base; Based on the time disclosure requirements in the confidentiality management strategy, and in conjunction with the time requirements, update the identification rules and / or mapping rules in the pre-built rule base.

[0012] In one embodiment, the step of tagging the unstructured data stored in the two-dimensional table according to the mapping rules and the preset data classification and grading list to obtain the tagging results includes: Load the mapping rules from the rule base; Based on the mapping rules, a matching analysis is performed on the unstructured data stored in the two-dimensional table; Based on the matching results and the preset data classification and grading list, the corresponding data category and data level are determined; The data category and data level are written as labels into the corresponding fields of the two-dimensional table to form the labeling result.

[0013] Furthermore, to achieve the above objectives, this application also proposes a data processing apparatus, the data processing apparatus comprising: The acquisition module is used to acquire unstructured data; The writing module is used to write the unstructured data into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list; The analysis module is used to analyze the correlation between the unstructured data and the preset data classification and grading list, and generate mapping rules; The tagging module is used to tag the unstructured data stored in the two-dimensional table according to the mapping rules and the preset data classification and grading list, and obtain the tagging result, wherein the tagging result includes the data category and data level corresponding to the unstructured data.

[0014] In addition, to achieve the above objectives, this application also proposes a data processing apparatus, the apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data processing method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the data processing method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data processing method described above.

[0017] This application proposes a data processing method, apparatus, device, storage medium, and program product. The method includes: acquiring unstructured data; writing the unstructured data into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list; analyzing the association between the unstructured data and the preset data classification and grading list to generate mapping rules; and labeling the unstructured data stored in the two-dimensional table according to the mapping rules and the preset data classification and grading list to obtain a labeling result, wherein the labeling result includes the data category and data level corresponding to the unstructured data. This method achieves effective classification and grading of unstructured data by constructing a two-dimensional table based on a preset classification and grading list and combining it with mapping rules to label records; simultaneously, it improves the accuracy of classification and grading by employing mapping rules that integrate keywords, regular expressions, and semantic recognition. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the data processing method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the data processing method of this application; Figure 3 This is a flowchart illustrating Embodiment 3 of the data processing method of this application; Figure 4 A simplified flowchart illustrating the data processing method provided in Embodiment 1 of this application; Figure 5 This is a schematic diagram of the module structure of the data processing device according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is: to obtain unstructured data; to write the unstructured data into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list; to analyze the relationship between the unstructured data and the preset data classification and grading list and generate mapping rules; and to label the unstructured data stored in the two-dimensional table according to the mapping rules and the preset data classification and grading list to obtain a labeling result, wherein the labeling result includes the data category and data level corresponding to the unstructured data.

[0025] In this embodiment, for ease of description, the following description will focus on the data processing system as the execution subject.

[0026] Existing data classification and grading methods primarily rely on structured data, focusing on organizing and managing data with standardized formats and clearly defined fields. This bias makes unstructured documents easily marginalized during the classification and grading process, resulting in incomplete data coverage. More importantly, some core and important data may therefore remain unmanaged or overlooked, creating potential security risks and posing hidden dangers to government information security.

[0027] This application provides a solution that automatically labels unstructured data by writing it into a structured two-dimensional table based on a preset data classification and grading list, and then using mapping rules consisting of keyword matching rules, regular expression rules, and semantic recognition rules. This achieves comprehensive, accurate, and traceable classification and grading management of unstructured data. This solution not only expands the scope of data governance but also significantly improves the effectiveness and accuracy of unstructured data classification and grading through rule-driven and structured storage mechanisms, effectively supporting compliance requirements and dynamic security control.

[0028] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions. The following description uses a personal computer as an example to illustrate this embodiment and the subsequent embodiments.

[0029] Based on this, embodiments of this application provide a data processing method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data processing method of this application.

[0030] In this embodiment, the data processing method includes steps S10 to S40: Step S10: Obtain unstructured data; It should be noted that unstructured data refers to data that does not have a predefined data model or fixed format, and whose content cannot be directly stored in a traditional relational database in a row and column format. This includes, but is not limited to, text-based electronic documents such as policy documents, internal requests, work plans, meeting minutes, contract texts, audit reports, email bodies, PDF scans, and Word documents.

[0031] In this embodiment, for the large amount of unstructured documents generated in asset management operations, the system first filters and collects relevant documents based on a document importance assessment mechanism (e.g., preliminary assessment based on issuing unit, security classification, business relevance, historical usage frequency, etc.). Subsequently, through file import, API interfaces, or batch synchronization, these documents are uniformly connected to the data lake within the big data infrastructure platform for centralized storage and management. Thus, the system can read and load target documents from this data lake as needed as input sources for subsequent classification and grading, thereby completing the acquisition of unstructured data.

[0032] Step S20: Write the unstructured data into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list; It should be noted that the two-dimensional table refers to a structured data storage format. Its table structure (including field names, field types, and field semantics) is not arbitrarily defined, but is strictly designed in advance according to governance requirements. It is used to carry metadata and content fragments extracted or mapped from unstructured source files (such as policy documents, request materials, and plan texts).

[0033] The data classification and grading list is the core quality basis of this embodiment. It is customized according to the actual asset management business and divides the primary data categories into: investment and operation data, institutional management data, technical management data, market information data, etc. On this basis, the data classification and grading list further clarifies the grading standards for each type of data (sensitivity levels such as core, important, general level 3, general level 2, and general level 1), and defines corresponding data characteristics for each (category, level) combination, such as: field name, data format, and associated business scenarios.

[0034] Understandably, since unstructured data itself lacks a unified format and field boundaries, directly performing rule matching or tagging will face problems such as low recognition efficiency, unreliable results, and difficulty in integrating into the existing data governance system. Therefore, step S20 is executed to first organize the unstructured data into two-dimensional table records according to the structure model guided by the checklist. This not only provides standardized input for subsequent automated recognition, but also makes the classification and grading results storable, queryable, and auditable, thereby achieving effective integration of unstructured data with the structured governance framework.

[0035] In one feasible embodiment, step S20 may include steps S21 to S24: Step S21: Determine the field information and corresponding field types of the two-dimensional table according to the data categories, grading standards and corresponding data characteristics defined in the preset data classification and grading list; In this embodiment, based on a preset data classification and grading list, the data categories, grading standards and corresponding data features defined therein are parsed, and the field information and corresponding field types of the two-dimensional table used to store the identification results of unstructured data are determined accordingly.

[0036] The determined fields include, but are not limited to: Data ID: Used to uniquely identify each record; the field type is string (VARCHAR). Data Category: Indicates the data category to which this record belongs; the field type is string. Data level: Indicates the sensitivity level of this record; the field type is string. Data content: Used to store raw text fragments (such as ID card numbers, contract amounts, project numbers, etc.) identified from unstructured source files, with the field type being long text (TEXT); Source File ID: A unique identifier in the data lake for the original unstructured document corresponding to this record; the field type is string. Recognition Time: Used to record the timestamp when the data was recognized; the field type is DateTime.

[0037] The design of the above fields is directly derived from the governance elements in the data classification and grading list, ensuring that the two-dimensional table structure is consistent with compliance requirements.

[0038] Step S22: Construct a two-dimensional table based on the field information and field types; In this embodiment, the system calls the table creation interface of the relational database and executes Structured Query Language (SQL) to dynamically create the target two-dimensional table based on the field name, field type and constraints determined in step S21.

[0039] Step S23: Perform data cleaning on the unstructured data to remove duplicate data and correct data with format errors, to obtain cleaned data; In this embodiment, before writing unstructured data into the two-dimensional table, a data cleaning operation is first performed to improve the accuracy and reliability of subsequent classification and grading. Specifically, based on the combination of data content and source file ID, duplicate records are identified and removed to prevent the same data fragment from being tagged multiple times. At the same time, the legality verification and standardization transformation of field fragments with structured semantics (such as date, amount, and ID number) identified from the unstructured data are performed. In addition, records with empty key fields (such as data category and source file ID) are marked or filtered to ensure that the data written into the two-dimensional table has complete metadata.

[0040] After the above processing, the cleaned data is output as the input source for ETL loading.

[0041] Step S24: The cleaned data is written into the two-dimensional table using a database data extraction, transformation, and loading ETL tool.

[0042] The ETL (Extract-Transform-Load) tool is a software tool used to automate the migration and processing of data from a source system to a target database.

[0043] In this embodiment, the system uses the open-source ETL tool Kettle (i.e., Pentaho Data Integration) to perform the data loading task. Specifically, firstly, a job workflow is created in Kettle, configuring "text file input" or "database query" as the data source, and accessing the data cleaned in step S23; then, through components such as "field selection" and "type conversion," it is ensured that the data type of each field is strictly consistent with the field definition of the target two-dimensional table; finally, the "table output" step is used to write the data in batches into the two-dimensional table constructed in step S22, and the transaction commit mechanism is enabled to ensure the atomicity and efficiency of the write.

[0044] In addition, after the data is written, indexes are created on the target table: including creating indexes on data category fields to support fast filtering of data categories, and creating indexes on data level fields to accelerate aggregate queries based on sensitivity levels, thereby improving the retrieval and auditing of subsequent tagging results.

[0045] Through the above steps, unstructured data is systematically transformed into structured records and stored in a standardized format as a two-dimensional table aligned with the data classification and grading list. This two-dimensional table not only possesses clear field semantics and type constraints, but also ensures content accuracy through data cleaning, and achieves efficient write and query capabilities through ETL loading and index building, thus providing a high-quality and manageable data foundation for subsequent tagging based on mapping rules.

[0046] Step S30: Analyze the relationship between the unstructured data and the preset data classification and grading list, and generate mapping rules; The mapping rules include, but are not limited to: Keyword rules: Based on the data characteristics defined in the data classification and grading list, extract distinctive keywords or phrases as the matching criteria. For example, if the list specifies that "documents containing 'ID card' and 'social security number' belong to core-level customer information," then words such as "ID card" and "social security number" will be included in the keyword rules. Regular expression rules: For data features with fixed formats (such as ID numbers, bank card numbers, dates, amounts, etc.), regular expressions are constructed for precise matching. For example, regular expressions can be used to identify ID card numbers; Semantic recognition rules: For complex contexts that cannot be accurately covered by keywords or regular expressions, natural language processing models are used to understand the semantics of the context and determine the data category and level.

[0047] Understandably, due to the diverse content and flexible expression of unstructured data, relying solely on manual annotation or a single recognition method can easily lead to missed or incorrect judgments, and is difficult to adapt to the needs of large-scale document processing. Therefore, step S30 is executed to systematically establish an automated mapping relationship between unstructured data content and the governance list by integrating multimodal rules such as keywords, regular expressions, and semantic recognition. This avoids the problems of incomplete rule coverage, low recognition accuracy, and high manual costs in traditional methods, thereby achieving efficient, accurate, and scalable classification and grading capabilities for unstructured data.

[0048] In one feasible embodiment, step S30 may include steps S31-S32: Step S31: For each data category and corresponding grading standard in the preset classification and grading list, determine the identification rules for identifying the unstructured data. In this step, the system iterates through the data categories and grading standards in the pre-defined classification and grading list, and generates appropriate recognition rules based on the defined data characteristics. For example, for the entry "Market Information Data - General Level 2" in the list, its data characteristic description is: "The document name or content contains keywords such as industry research reports, market analysis, and competitor dynamics, but does not involve internal strategies or customer details." Based on this, the system can configure the following recognition rules: Keyword rules: Match whether the document name or body text contains words such as "industry research report", "market trend", or "competitive analysis"; Regular expression rules: Identify whether it contains typical market data formats, such as "\d{4}year Q[1-4]market size"; Semantic recognition rules: Invoke a pre-trained text classification model to determine whether the overall semantics of the document fall within the scope of public market information, rather than internal sensitive content.

[0049] Using the above method, for each (category, level) pair in the list, the system automatically generates one or more complementary recognition rules to ensure coverage of explicit keywords, structured patterns, and implicit semantic scenarios.

[0050] Step S32: Associate the identification rules with the corresponding data categories and grading standards to form the mapping rules.

[0051] In this step, the system structurally binds the recognition rules generated in step S31 to the data categories and data levels from which they originate, forming complete mapping rules. Specifically, each mapping rule contains four core elements: a data category field, used to identify the business category to which the rule applies; a data level field, used to identify the corresponding security sensitivity level; a rule type field, used to describe the recognition method used (such as keyword rules, regular expression rules, or semantic recognition rules); and a rule content field, used to store the specific matching conditions or recognition logic.

[0052] For example, in the typical scenario of "logistics management system," it is defined as "organizational management data" in the preset data classification and grading list, with a security level of "general level 1." Based on this, the system generates corresponding mapping rules: when the document name or document content contains keywords such as "logistics," "canteen," or "vehicle management," or matches a fixed text pattern such as "reimbursement standard: AA yuan / day," the mapping rule is triggered, and the relevant unstructured data is classified as "organizational management data - general level 1."

[0053] Through the above steps, the mapping rules achieve precise alignment between governance requirements and recognition logic, providing clear and executable judgment criteria for subsequent automated labeling.

[0054] Step S40: According to the mapping rules and the preset data classification and grading list, the unstructured data stored in the two-dimensional table is tagged to obtain the tagging result, wherein the tagging result includes the data category and data level corresponding to the unstructured data.

[0055] Understandably, since structured data itself lacks explicit category and level identifiers, relying on manual labeling of each item is not only inefficient but also susceptible to subjective judgment, making it difficult to guarantee consistency and compliance. Therefore, step S40 is executed to automatically match mapping rules and combine them with a preset data classification list for labeling, avoiding deviations and omissions caused by manual intervention, thereby achieving efficient, consistent, and auditable classification results.

[0056] In one feasible embodiment, step S40 may include steps S41 to S44: Step S41: Load the mapping rule from the rule base; In this step, when starting the labeling task, all mapping rules are first read from a pre-built rule base. This rule base stores the recognition logic generated in step S32, aligned with the data classification and grading list, ensuring that the rules used in the labeling process are complete, up-to-date, and controlled.

[0057] Step S42: Based on the mapping rules, perform matching analysis on the unstructured data stored in the two-dimensional table; In this step, the system iterates through each record in the two-dimensional table, and for its "data content" field (i.e., the text fragment extracted from the original document) and the document metadata associated with the "source file ID" (such as the filename), it sequentially applies keyword rules, regular expression rules, and semantic recognition rules from the rule base for matching. For example, if the data content of a record contains "canteen management rules", the keyword rule is triggered; if it contains "travel reimbursement standard: 500 yuan / day", the regular expression rule is matched.

[0058] Step S43: Determine the corresponding data category and data level based on the matching results and the preset data classification and grading list; In this step, the data category and data level associated with the successfully matched mapping rules are used as candidate results. If multiple rules match the same record, the final data category and level are determined according to a preset priority strategy (such as "core > important > general" or rule confidence ranking) to ensure that the tagging results meet the governance requirements defined in the checklist.

[0059] Step S44: Write the data category and data level as labels into the corresponding fields of the two-dimensional table to form the labeling result.

[0060] Finally, the data categories determined in step S43 are written into the data category field of the two-dimensional table, and the data levels are written into the data level field, completing the structured labeling. After labeling is completed, the complete labeling results (including data ID, data category, data level, data content, source file ID, identification time, etc.) are synchronously stored in the pre-built labeling result table to support subsequent governance operations such as data asset inventory, security policy enforcement, access control, compliance auditing, and visualization.

[0061] Through the above steps, not only are unstructured data given clear classification and hierarchical attributes, but they are also incorporated into a unified data asset catalog in a structured form, achieving a governance leap from "unmanageable" to "identifiable, controllable, and traceable".

[0062] The method described above obtains unstructured data; writes the unstructured data into a two-dimensional table, which is determined based on a preset data classification and grading list; analyzes the relationship between the unstructured data and the preset data classification and grading list to generate mapping rules; and labels the unstructured data stored in the two-dimensional table according to the mapping rules and the preset data classification and grading list to obtain labeling results, wherein the labeling results include the data category and data level corresponding to the unstructured data. This method achieves effective classification and grading of unstructured data by constructing a two-dimensional table based on a preset classification and grading list and combining it with mapping rules to label records; simultaneously, it improves the accuracy of classification and grading by employing mapping rules that integrate keywords, regular expressions, and semantic recognition.

[0063] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 After step S24, the data processing method further includes step S241: Step S241: Sampling inspection is performed on the data in the two-dimensional table to verify the matching degree of the data with the preset data classification and grading list, the integrity of the table structure, and the consistency of the data.

[0064] Compared to the first embodiment, this embodiment also proposes a data quality verification mechanism. After the cleaned data is written into a two-dimensional table in batches, data verification is performed. This mechanism can promptly detect and locate labeling deviations or structural defects caused by rule mismatch, ETL anomalies, or insufficient cleaning, thereby ensuring the accuracy, compliance, and auditability of the classification and grading results.

[0065] Specifically, verification samples are selected from the already written two-dimensional table according to a preset sampling strategy. This sampling strategy includes, but is not limited to: stratified sampling by data category, proportional sampling by data level, or full or high-proportion coverage sampling of core / important data. Subsequently, the data category and data level corresponding to each extracted record are back-compare with its original data content and the applied mapping rules to confirm whether its labeling results strictly conform to the definitions in the preset data classification and grading list. For example, if a record is labeled "Customer Information - Core," its data content should contain core features explicitly defined in the list, such as ID card number and bank account number, and the triggered mapping rule should indeed belong to this (category, level) combination.

[0066] Next, check each required field in the two-dimensional table for empty values, incorrect types, or abnormal formats. For example, the "Identification Time" field should be in a valid date and time format, the "Data ID" and "Source File ID" should not be empty, and the values ​​of "Data Category" and "Data Level" should be within the range allowed by the list.

[0067] Finally, for multiple records originating from the same unstructured document, verify whether their labeling results are consistent within the same business context. For example, the "contract amount" fragment appearing multiple times in the same contract document should be assigned the same category (e.g., "financial data") and level (e.g., "important") to avoid classification conflicts caused by minor differences in context.

[0068] If inconsistencies, omissions, or logical conflicts are found during the verification process, a data quality report is generated, abnormal records are marked, and manual review or feedback to the rule base is supported for optimization and iteration.

[0069] By using the methods described above, after writing the two-dimensional table, data verification is performed. By sampling and checking, the matching degree between the data and the list, the integrity of the table structure, and the consistency of the data are confirmed. This can effectively verify whether the classification and grading results meet the preset governance requirements, and promptly identify labeling deviations or structural omissions caused by rule configuration errors, ETL processing anomalies, or insufficient data cleaning. This ensures the accuracy, compliance, and auditability of the unstructured data governance process.

[0070] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After step S32, the data processing method further includes steps S321 to S322: Step S321: Store the identification rules and the mapping rules in a pre-built rule base; Step S322: Based on the time disclosure requirements in the confidentiality management strategy, and in conjunction with the time requirements, update the identification rules and / or mapping rules in the pre-built rule base.

[0071] Compared to the first embodiment, this embodiment also proposes a dynamic rule update mechanism oriented towards timeliness and compliance requirements. This mechanism enables the classification and grading rules to automatically evolve with the time constraints of confidentiality policies, ensuring that the grading results of unstructured data always meet the latest security control requirements and avoiding the risk of compliance lag caused by rule solidification.

[0072] Specifically, the identification rules generated in steps S31-S32, along with their associated data categories, data levels, and other information, are first persistently stored in a pre-built rule base in a structured format. This rule base supports efficient querying and version management, providing a unified rule source for subsequent tagging.

[0073] Upon receiving a notification of a change in confidentiality policy, the system parses the clauses regarding time-based disclosure requirements in the confidentiality management regulations (e.g., "Internal plans involving major projects will be downgraded to Core → Important within 6 months of project disclosure," and "Annual audit reports can be adjusted from 'Core' to 'General Level 2' after March 1st of the following year") and automatically adjusts the relevant mapping rules in the rule base accordingly. For example, it can attach an expiration date attribute to mapping rules of specific categories, or trigger rule content updates (such as downgrading data levels or relaxing keyword matching conditions) after a specified time node is reached. By embedding time elements into rule lifecycle management, this embodiment achieves adaptive responses of classification and grading capabilities to dynamic compliance environments.

[0074] By incorporating the identification and mapping rules into a manageable rule base and dynamically updating the rules in accordance with the time disclosure requirements in confidentiality management, the data classification and grading capabilities can automatically evolve with the timeliness of policies. This effectively solves the problems of slow response, high manual maintenance costs, and easy security level mismatch in the face of dynamic compliance scenarios in traditional static rule systems, thereby improving the timeliness, compliance and automation level of unstructured data governance.

[0075] For example, to help understand the implementation flow of the data processing method obtained by combining this embodiment with the first embodiment described above, please refer to... Figure 4 , Figure 4 A simplified flowchart of a data processing method is provided, specifically: The method in this embodiment includes four main stages: data preparation stage, two-dimensional table design and storage stage, unstructured data processing stage, and tagging implementation and storage stage.

[0076] In the data preparation phase, unstructured documents (such as policy documents, request materials, and plan texts) from various business systems are first connected to the big data platform. Then, a data classification and grading list is developed to clarify the categories, levels, and corresponding data characteristics of various types of data. At the same time, based on the time disclosure requirements in confidentiality management, the identification rules and mapping rules in the rule base are regularly synchronized and updated to ensure the timeliness of the governance strategy.

[0077] In the two-dimensional table design and storage phase, the two-dimensional table structure is designed based on the data classification and hierarchical list, including field names, field types, and semantic definitions; the extracted unstructured data is cleaned to remove duplicate records and correct format errors; the cleaned data is written to the two-dimensional table in batches using ETL tools (such as Kettle), and indexes are created to improve query efficiency; finally, the written data is sampled and checked to verify its matching degree with the list, table structure integrity, and data consistency.

[0078] In the unstructured data processing stage, the system analyzes the relationship between unstructured data and classification and grading lists, generates identification rules of the type such as keywords, regular expressions or semantic recognition, and associates them with the corresponding data categories and levels to form mapping rules; then these mapping rules are stored in the rule base as the basis for subsequent labeling decisions.

[0079] During the tagging implementation and storage phase, the system loads mapping rules from the rule base, performs matching analysis on the unstructured data stored in the two-dimensional table, determines the data category and data level based on the matching results, and writes this information as a label into the corresponding field of the two-dimensional table to complete automatic tagging. Finally, the tagging results are stored uniformly to form structured data assets that can be used for auditing, authorization, and asset management.

[0080] Through the methods described in the above embodiments, this application realizes a closed-loop governance system covering the entire chain from unstructured data access to automated classification and grading, possessing high accuracy, strong compliance, and good scalability.

[0081] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data processing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0082] This application also provides a data processing apparatus, please refer to... Figure 5 The data processing device includes: The acquisition module is used to acquire unstructured data; The writing module is used to write the unstructured data into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list; The analysis module is used to analyze the correlation between the unstructured data and the preset data classification and grading list, and generate mapping rules; The tagging module is used to tag the unstructured data stored in the two-dimensional table according to the mapping rules and the preset data classification and grading list, and obtain the tagging result, wherein the tagging result includes the data category and data level corresponding to the unstructured data.

[0083] The data processing apparatus provided in this application, employing the data processing method described in the above embodiments, can solve the technical problem of how to accurately and effectively classify and grade unstructured data. Compared with the prior art, the beneficial effects of the data processing apparatus provided in this application are the same as those of the data processing method described in the above embodiments, and other technical features in the data processing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0084] This application provides a data processing apparatus, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method in Embodiment 1 above.

[0085] The following is for reference. Figure 6 The diagram illustrates a structural schematic of a data processing device suitable for implementing embodiments of this application. The data processing device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The data processing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0086] like Figure 6 As shown, the data processing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the data processing device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the data processing device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show data processing devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0087] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0088] The data processing device provided in this application, employing the data processing method described in the above embodiments, can solve the technical problem of how to accurately and effectively classify and grade unstructured data. Compared with the prior art, the beneficial effects of the data processing device provided in this application are the same as those of the data processing method described in the above embodiments, and other technical features of this data processing device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0089] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0090] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0091] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data processing method described in the above embodiments.

[0092] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0093] The aforementioned computer-readable storage medium may be included in a data processing device or may exist independently without being assembled into a data processing device.

[0094] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a data processing device, the data processing device causes the data processing device to: acquire unstructured data; write the unstructured data into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list; analyze the association between the unstructured data and the preset data classification and grading list, and generate mapping rules; and, according to the mapping rules and the preset data classification and grading list, label the unstructured data stored in the two-dimensional table to obtain a labeling result, wherein the labeling result includes the data category and data level corresponding to the unstructured data.

[0095] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0097] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0098] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described data processing method, and can solve the technical problem of how to accurately and effectively classify and grade unstructured data. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data processing method provided in the above embodiments, and will not be repeated here.

[0099] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method described above.

[0100] The computer program product provided in this application can solve the technical problem of how to accurately and effectively classify and grade unstructured data. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the data processing method provided in the above embodiments, and will not be repeated here.

[0101] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A data processing method, characterized in that, The data processing method includes: Acquire unstructured data; The unstructured data is written into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list; Analyze the relationship between the unstructured data and the preset data classification and grading list to generate mapping rules; According to the mapping rules and the preset data classification and grading list, the unstructured data stored in the two-dimensional table is tagged to obtain the tagging result, wherein the tagging result includes the data category and data level corresponding to the unstructured data.

2. The data processing method as described in claim 1, characterized in that, The step of writing the unstructured data into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list, includes: Based on the data categories, grading standards, and corresponding data characteristics defined in the preset data classification and grading list, determine the field information and corresponding field types of the two-dimensional table; A two-dimensional table is constructed based on the field information and field types; The unstructured data is cleaned to remove duplicate data and correct data with format errors, resulting in cleaned data. The cleaned data is written into the two-dimensional table using database data extraction, transformation, and loading ETL tools.

3. The data processing method as described in claim 2, characterized in that, After the step of writing the cleaned data into the two-dimensional table using database data extraction, transformation, and loading ETL tools, the method further includes: The data in the two-dimensional table is sampled and checked to verify the matching degree between the data and the preset data classification and grading list, the integrity of the table structure, and the consistency of the data.

4. The data processing method as described in claim 1, characterized in that, The step of analyzing the correlation between the unstructured data and the preset data classification and grading list, and generating mapping rules, includes: For each data category and corresponding grading standard in the preset classification and grading list, identification rules for identifying the unstructured data are determined, wherein the identification rules include one or more of keyword rules, regular expression rules, or semantic recognition rules; The identification rules are associated with the corresponding data categories and grading standards to form the mapping rules.

5. The data processing method as described in claim 4, characterized in that, After the step of associating the identification rules with the corresponding data categories and grading standards to form the mapping rules, the method further includes: The identification rules and the mapping rules are stored in a pre-built rule base; Based on the time disclosure requirements in the confidentiality management strategy, and in conjunction with the time requirements, update the identification rules and / or mapping rules in the pre-built rule base.

6. The data processing method as described in claim 5, characterized in that, The step of tagging the unstructured data stored in the two-dimensional table according to the mapping rules and the preset data classification and grading list to obtain the tagging results includes: Load the mapping rules from the rule base; Based on the mapping rules, a matching analysis is performed on the unstructured data stored in the two-dimensional table; Based on the matching results and the preset data classification and grading list, the corresponding data category and data level are determined; The data category and data level are written as labels into the corresponding fields of the two-dimensional table to form the labeling result.

7. A data processing apparatus, characterized in that, The data processing device includes: The acquisition module is used to acquire unstructured data; The writing module is used to write the unstructured data into a two-dimensional table, wherein the two-dimensional table is determined based on a preset data classification and grading list; The analysis module is used to analyze the correlation between the unstructured data and the preset data classification and grading list, and generate mapping rules; The tagging module is used to tag the unstructured data stored in the two-dimensional table according to the mapping rules and the preset data classification and grading list, and obtain the tagging result, wherein the tagging result includes the data category and data level corresponding to the unstructured data.

8. A data processing device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data processing method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data processing method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 6.

Images