Data management method and device based on field classification, equipment, medium and program product
By extracting features from database table structure fields and assigning classification models, the problem of low efficiency in manual evaluation in data storage management is solved, achieving efficient utilization of storage resources and improved query performance.
Patent Information
- Application Number
- CN202511363994.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-23
- Publication Date
- 2026-01-13
Smart Images
Figure CN121326902A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence, to the application of large models in fintech scenarios, and more specifically to a data management method, device, equipment, medium, and program product based on field classification. Background Technology
[0002] Current data storage management largely relies on administrators manually assessing the access patterns and importance of data tables, and then manually configuring storage partitions. With the surge in data volume, this manual approach is inefficient and struggles to adapt to dynamic changes in data access hotspots in real time. This often results in high-performance storage resources being underutilized or low-performance storage impacting the query speed of critical business operations. Summary of the Invention
[0003] In view of the above problems, this application provides a data management method, apparatus, device, medium and program product based on field classification.
[0004] According to a first aspect of this application, a data management method based on field classification is provided, comprising: acquiring field information of a database table structure to be processed; extracting features from the field information to be detected to obtain numerical features representing field semantics and field context information; inputting the numerical features into a pre-trained data classification model to obtain classification results for data categories and information levels corresponding to the field information; and allocating fields with different classification results to different storage partitions of the database based on the classification results for data categories and information levels. The pre-trained data classification model is obtained by generating an initial data classification model from a pre-constructed initial database, generating an optimized dataset based on the business scenario context information of the database table, and optimizing and training the initial data classification model based on the optimized dataset.
[0005] According to an embodiment of this application, the field information includes a field name, data type, table name, and table usage description. The step of extracting features from the field information to be detected to obtain numerical features representing the field semantics and field context information includes: performing word segmentation and word vectorization on the field name to generate a field name semantic feature vector; performing natural language processing on the table name and table usage description to generate a table context semantic feature vector; converting the data type into a numerical feature; and fusing the field name semantic feature vector, the table context semantic feature vector, and the data type numerical feature to obtain the numerical feature.
[0006] According to an embodiment of this application, the method further includes: obtaining the first batch of labeled data from historical manual annotation records, wherein the first batch of labeled data consists of field information that has been manually annotated and the data category label and information level label corresponding to the field information; extracting the second batch of labeled data from a preset data processing standard table, wherein the second batch of labeled data includes standard field information and the standard data category label and information level label corresponding to the standard field information; and merging the first batch of labeled data and the second batch of labeled data to form the initial database.
[0007] According to an embodiment of this application, generating an optimized dataset based on business scenario context information from a database table includes: obtaining fields whose semantics or information level changes under a specified business scenario; determining the scenario purpose of the table to which the field belongs; and re-labeling the data category label and information level label corresponding to the field according to the scenario purpose of the table to which the field belongs, thereby obtaining the optimized dataset.
[0008] According to an embodiment of this application, after inputting the numerical features into a pre-trained data classification model to obtain the classification results of the data category and information level corresponding to the field information, the method further includes: matching the field information to be detected with the rules in the benchmark rule base; if the match is successful, the classification result determined by the benchmark rule base is used as the final classification result of the field; if the match is unsuccessful, the classification result output by the pre-trained data classification model is used as the final classification result of the field.
[0009] According to an embodiment of this application, the method includes: receiving a user's correction instruction for the classification result; generating new labeled data based on the correction instruction, and adding the new labeled data to the initial database to obtain an updated initial database; and iteratively training the data classification model using the updated initial database.
[0010] According to an embodiment of this application, after inputting the numerical features into a pre-trained data classification model to obtain the classification results of the data category and information level corresponding to the field information, the method further includes: outputting the confidence score corresponding to the classification result; if the confidence score is higher than a first threshold, performing the operation of matching the field information to be detected with the rules in a preset benchmark rule base; if the confidence score is lower than the first threshold but higher than a second threshold, marking the classification result with a first mark and pushing it to the manual review interface, wherein the first mark indicates that the review status of the classification result is pending review; if the confidence score is lower than the second threshold, marking the field to be detected with a second mark and pushing it to the manual review interface, wherein the second mark indicates that the classification result of the field to be detected cannot be determined.
[0011] A second aspect of this application provides a data management device based on field classification. The device includes: an acquisition module for acquiring field information of a database table structure to be processed; a feature extraction module for extracting features from the field information to be detected to obtain numerical features representing field semantics and field context information; an input module for inputting the numerical features into a pre-trained data classification model to obtain classification results of data category and information level corresponding to the field information, wherein the pre-trained data classification model generates an initial data classification model through a pre-constructed initial database, generates an optimized dataset based on the business scenario context information of the database table, and optimizes and trains the initial data classification model based on the optimized dataset; and an allocation module for allocating fields of different classification results to different storage partitions of the database based on the classification results of data category and information level.
[0012] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.
[0013] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.
[0014] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description
[0015] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:
[0016] Figure 1 The illustrations depict application scenarios of data management methods, apparatuses, devices, media, and program products based on field classification according to embodiments of this application.
[0017] Figure 2 A flowchart illustrating a data management method based on field classification according to an embodiment of this application is shown schematically.
[0018] Figure 3 This schematically illustrates a structural block diagram of a data management device based on field classification according to an embodiment of this application; and
[0019] Figure 4A block diagram schematically illustrates an electronic device suitable for implementing a field-based data management method according to an embodiment of this application. Detailed Implementation
[0020] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.
[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0022] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0023] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).
[0024] In the technical solution of this application, the user information (including but not limited to user personal information, user cross-border remittance information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, application, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.
[0025] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.
[0026] Figure 1 The illustration shows an application scenario diagram of a data management method, apparatus, device, medium, and program product based on field classification according to embodiments of this application.
[0027] like Figure 1 As shown, Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0028] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).
[0029] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0030] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.
[0031] It should be noted that the data management method based on field classification provided in this application embodiment can generally be executed by server 105. Correspondingly, the data management device based on field classification provided in this application embodiment can generally be located in server 105. The data management method based on field classification provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data management device based on field classification provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.
[0032] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0033] Figure 2 A flowchart illustrating a data management method based on field classification according to an embodiment of this application is shown schematically.
[0034] like Figure 2 As shown, the data management method based on field classification in this embodiment includes operations S210 to S240.
[0035] In operation S210, obtain the database table structure field information to be processed.
[0036] According to embodiments of this application, field information refers to various metadata information related to the field to be detected. Metadata is data that describes data, such as field name, data type, data length, name of the table to which it belongs, and a description of the purpose of the table, used to understand the actual meaning and data characteristics represented by the field.
[0037] According to embodiments of this application, the metadata information can be obtained from a database table. A database connection is established, query commands are executed, and the returned result set is received by calling the application programming interface and driver provided by the database management system. The required field information is then parsed from the result set.
[0038] In operation S220, feature extraction is performed on the field information to obtain numerical features that represent the semantics and contextual information of the fields.
[0039] According to embodiments of this application, numerical features can be features represented in numerical form after feature extraction. These numerical features can quantitatively describe the semantics and contextual information of a field. For example, vectors can be used to represent the features of a field, where each dimension of the vector corresponds to a specific feature attribute, and its numerical value reflects the strength or importance of that attribute in the field.
[0040] In operation S230, numerical features are input into the pre-trained data classification model to obtain the classification results of data categories and information levels corresponding to field information.
[0041] According to an embodiment of this application, the pre-trained data classification model is a model that has been built through a specific training process. It is used to classify input numerical features and determine the data category and information level corresponding to field information. This data classification model is pre-trained through the following process: constructing an initial database; generating an initial data classification model from the constructed initial database; generating an optimized dataset based on the business scenario context information of the database tables; and optimizing and training the initial data classification model based on the optimized dataset to obtain the pre-trained data classification model. This model has the ability to classify new input numerical features.
[0042] According to embodiments of this application, the initial database can be a basic dataset used to train a data classification model. This database contains labeled field information, data category labels, and information level labels. The labeled field information can be field metadata with relevant attributes manually or otherwise labeled; the data category labels indicate the category to which the field belongs; and the information level labels identify the level of the field.
[0043] According to embodiments of this application, the business context information of a database table can refer to metadata describing the specific purpose and importance of a database table in a business system. The business context information of a database table can include the table's business functions and the nature of its data. It is understood that even if two fields have the same name, their data categories and information levels can differ when they appear in tables within different business scenarios.
[0044] In operation S240, based on the classification results of data category and information level, fields of different classification results are assigned to different storage partitions of the database.
[0045] According to embodiments of this disclosure, different storage areas can be partitioned within a database according to certain rules. Based on the classification results of data categories and information levels, fields of different categories are assigned to different storage partitions, enabling classified storage and management of data and improving data query efficiency, security, and maintainability. For example, fields with specific information can be stored in partitions subject to strict access control, while frequently queried fields can be stored in partitions with higher performance.
[0046] According to the embodiments of this application, the semantics and context of database fields are parsed to classify massive table structure fields. Based on the classification results, fields of different categories are allocated to storage partitions with matching performance, thereby realizing refined and automated management of database storage resources. This can improve the query and access performance of the database system while effectively reducing storage hardware costs.
[0047] In some embodiments of this application, field names can be segmented and vectorized to generate semantic feature vectors for field names; natural language processing can be performed on table names and table usage descriptions to generate semantic feature vectors for table context; data types can be converted into numerical features to generate numerical features for data types; and the semantic feature vectors for field names, semantic feature vectors for table context, and numerical features for data types can be fused to form feature vectors.
[0048] Tokenization is the process of cutting a continuous string of field names into individual words or phrases according to certain rules. Word vectorization is the process of converting the segmented words into numerical vectors. By integrating the word vectors of all words in the field name, a feature vector representing the semantics of the entire field name can be obtained.
[0049] In some embodiments, the table name and table purpose description text can be concatenated into a complete description text, and the description text can be encoded using a sentence encoding model to directly generate a table context semantic feature vector representing the table context.
[0050] In some embodiments, a data type mapping table can be established to map different data types to unique numerical codes. The data type of a field can be used as a query key to find the corresponding numerical code from the data type mapping table in order to generate data type numerical characteristics.
[0051] In some embodiments, the semantic feature vector of field name, the semantic feature vector of table context, and the numerical feature of data type can be input into a feature fusion neural network. The feature fusion neural network calculates a weighted fused feature vector, which is then used as the feature vector input to the data classification model.
[0052] According to the embodiments of this application, by numerically fusing three heterogeneous features—field name semantics, table context semantics, and data type—a high-dimensional feature vector that can comprehensively and deeply represent field attributes and business scenarios is generated, significantly improving the accuracy and reliability of subsequent data classification.
[0053] In some embodiments of this application, the first batch of labeled data can be obtained from historical manual annotation records; the second batch of labeled data can be extracted from a preset data processing standard table; and the first batch of labeled data and the second batch of labeled data can be merged to form an initial database for training a data classification model.
[0054] The first batch of labeled data consists of manually annotated field information, along with corresponding data category labels and information level labels. This first batch of labeled data may also include at least one of the following: field name, data type, table name, and table usage description. The table usage description in the labeled data can be used to provide contextual semantic information for fields during model training, distinguishing the data type and information level of the same field in different business scenarios.
[0055] Historical manually labeled records are the data type and information level results that have been completed and manually reviewed during data processing. They reflect the judgment and experience regarding data type and information level in actual business scenarios, and can provide valuable samples for model training.
[0056] In some embodiments, historical manual annotation records are stored in the enterprise's metadata management system or data asset catalog.
[0057] In some embodiments, the data processing standard table can be obtained from industry standard reference documents, from which the extracted data can provide standardized samples for training the initial data classification model, helping the model learn common data types and information level determination rules.
[0058] In some embodiments, when merging the first batch of labeled data and the second batch of labeled data, the labeled data from different sources can be formatted and conflict checked. When data conflicts occur, the data classification results in the historical manual annotation records are used first.
[0059] According to the embodiments of this application, by merging two batches of labeled data obtained from historical manual annotation records and a pre-set standard table, the advantages of data from different sources can be integrated. Historical manual annotation data is specific to actual business scenarios, while the pre-set standard table data is standardized and universal. The merged initial database can provide richer and more comprehensive samples for model training, improving the model's generalization ability and classification accuracy.
[0060] In some embodiments of this application, fields whose semantics or information level changes under a specified business scenario can be obtained; the scenario purpose of the table to which the field belongs can be determined; and the data category label and information level label corresponding to the field can be re-labeled according to the scenario purpose of the table to which the field belongs, so as to obtain an optimized dataset.
[0061] A specified business scenario refers to a specific work context or scope of compliance requirements defined for a particular industry, business area, or application environment. In this scenario, the semantic meaning, purpose of use, data type, and information level of the data may differ from those in a general scenario.
[0062] According to the embodiments of this application, by combining the contextual information of specific business scenarios, fields whose data types and information levels change with the scenario are accurately identified, and targeted annotation and optimization are performed according to the actual use of the data table to which they belong, generating an optimized dataset that is highly aligned with business needs. This effectively solves the classification bias problem caused by the lack of scenario-based knowledge in general models, and significantly improves the accuracy, compliance and adaptability of data classification models in specific business environments.
[0063] In some embodiments of this application, the field information to be detected can be matched with the rules in the benchmark rule base; if the match is successful, the classification result determined by the benchmark rule base is used as the final classification result of the field; if the match is unsuccessful, the classification result output by the pre-trained data classification model is used as the final classification result of the field.
[0064] The rules in the benchmark rule base can be predefined logical judgment conditions that can be used to identify and classify data. For example, they can be rules based on keywords, rules based on regular expressions, and so on.
[0065] In one implementation, the field name in the field information can be matched against a predefined keyword list in a baseline rule base. If the field name contains any of the predefined keywords, the match is considered successful. For example, consider a field A_m to be detected, whose classification result is Class I Data (data type) - Level 1 (information level). The rule base contains a rule that the classification result corresponding to keyword m is Class I Data - Level 2. Comparing the field name A_m with the keyword list, if it contains keyword m, the match is successful, and the classification result defined in the rule base (Class I Data - Level 2) is forcibly adopted as the final result.
[0066] In one implementation, sample data under a field can be obtained; the sample data can be matched with regular expression rules in a benchmark rule base; if the sample data conforms to the defined pattern of any regular expression rule, it is determined that the match is successful.
[0067] For example, suppose there is a field C_h to be detected, and its classification result is Class I Data - Level 1. First, keyword matching is performed, but the field name h is not in the keyword list. Then, regular expression matching is performed, but the sample data under this field does not conform to the preset data format. Therefore, the matching fails, and the classification result (Class I Data - Level 1) is taken as the final classification result.
[0068] According to the embodiments of this application, by matching the information of the field to be detected with a predefined benchmark rule base, if the match is successful, the classification result determined by the rule base is adopted; if the match is unsuccessful, the classification result is retained. This dual verification mechanism not only ensures the accuracy and compliance of the classification by leveraging the explicit rules of the rule base, effectively avoiding misjudgments caused by model limitations, but also uses the model classification results to cover complex situations not covered by the rule base, improving the comprehensiveness and flexibility of data classification, thereby providing more reliable protection for the data.
[0069] In some embodiments of this application, a manual review interface can be provided to receive user correction instructions for the final classification results; new labeled data can be generated based on the correction instructions, and the new labeled data can be added to the initial database to obtain an updated initial database; the updated initial database can be used to iteratively train the data classification model.
[0070] The manual review interface can be a program interface or channel that provides users with interactive operations, allowing them to interact with the data classification system. Through this interface, users can view the generated final classification results and make judgments and actions based on the actual situation. Users can issue correction commands on the manual review interface, which are operation instructions used to change the final classification results. Correction commands typically include the user's new classification judgment information for field information, such as changing the information level label of a field from "Level 3" to "Level 5".
[0071] In some embodiments, final classification results with confidence levels below a preset threshold, or fields that are not matched in the benchmark rule base and are frequently viewed by users, can be prioritized for manual review.
[0072] In some embodiments, modifications made by users to the final data category and information level results of specific fields through a manual review interface can be recorded; the modified results are then bound to the corresponding field information to generate a new structured labeled data.
[0073] According to the embodiments of this application, a manual review interface is provided to introduce a manual correction mechanism, which can promptly correct the deviation of automatic classification and ensure that the classification results are accurate and consistent with reality; new labeled data is generated based on the correction instructions and the initial database is updated, providing richer and more accurate materials for model iterative training; the updated database is used to iteratively train the model, enabling the model to continuously learn and optimize, and continuously improve its classification ability and adaptability to diverse data.
[0074] In some embodiments of this application, the data processing model can also output the confidence level corresponding to the classification result. The confidence level can reflect the model's assessment of the reliability or accuracy of the classification result it provides. The confidence level value is usually between 0 and 1. The closer the value is to 1, the more confident the model is in the classification result.
[0075] If the confidence level is higher than the first threshold, then operation S240 is executed to verify the classification result and generate the final classification result. The first threshold is greater than the second threshold.
[0076] If the confidence level is below the first threshold but above the second threshold, the classification result is marked with a first flag and pushed to the manual review interface. The first flag indicates that the review status of the classification result is pending review, reminding the manual reviewers to further examine and confirm the result. This manual review intervention ensures the accuracy of the classification results. Simultaneously, the first flag helps manual reviewers quickly identify records requiring review, improving review efficiency.
[0077] If the confidence level is lower than the second threshold, the field to be detected is given a second label and pushed to the manual review interface. The second label indicates that the classification result of the field to be detected cannot be determined, and manual reviewers are required to conduct in-depth analysis and judgment. For complex or ambiguous cases that the model cannot handle, manual review provides more accurate and flexible classification decisions, ensuring that all fields can be classified reasonably and avoiding classification errors or omissions caused by model limitations.
[0078] Based on the above-described data management method based on field classification, this application also provides a data management device based on field classification. The following will be combined with... Figure 3 The device is described in detail.
[0079] Figure 3 A schematic block diagram of a data management device based on field classification according to an embodiment of this application is shown.
[0080] like Figure 3 As shown, the data management device based on field classification in this embodiment includes an acquisition module 310, a feature extraction module 320, an input module 330, and an allocation module 340.
[0081] The acquisition module 310 is used to acquire the database table structure field information to be processed. In one embodiment, the receiving module 310 can be used to perform the operation S210 described above, which will not be repeated here.
[0082] The feature extraction module 320 is used to extract features from the field information to be detected, obtaining numerical features that characterize the semantics and contextual information of the field. In one embodiment, the feature extraction module 320 can be used to perform the operation S220 described above, which will not be repeated here.
[0083] Input model 330 is used to input the numerical features into a pre-trained data classification model to obtain classification results of data categories and information levels corresponding to the field information. The pre-trained data classification model generates an initial data classification model through a pre-constructed initial database, generates an optimized dataset based on the business scenario context information of the database table, and optimizes and trains the initial data classification model based on the optimized dataset. In one embodiment, input module 330 can be used to perform the operation S230 described above, which will not be repeated here.
[0084] The allocation module 340 is used to allocate fields of different classification results to different storage partitions of the database based on the classification results of the data category and information level. In one embodiment, the allocation module 340 can be used to perform the operation S240 described above, which will not be repeated here.
[0085] According to an embodiment of this application, the field information includes a field name, data type, table name, and table usage description. The feature extraction module includes: a first generation submodule, used to perform word segmentation and word vectorization on the field name to generate a field name semantic feature vector; a second generation submodule, used to perform natural language processing on the table name and table usage description of the table to which the field belongs to generate a table context semantic feature vector; a third generation submodule, used to convert the data type of the field into a numerical feature; and a fusion submodule, used to fuse the field name semantic feature vector, the table context semantic feature vector, and the data type numerical feature to obtain the numerical feature.
[0086] According to an embodiment of this application, the apparatus further includes: a database construction module, configured to obtain the first batch of labeled data from historical manual annotation records, wherein the first batch of labeled data consists of field information that has been manually annotated and data category labels and information level labels corresponding to the field information; extract the second batch of labeled data from a preset data processing standard table, wherein the second batch of labeled data includes standard field information and standard data category labels and information level labels corresponding to the standard field information; and merge the first batch of labeled data and the second batch of labeled data to form the initial database.
[0087] According to an embodiment of this application, generating an optimized dataset based on business scenario context information includes: obtaining fields whose semantics or information level changes under a specified business scenario; determining the scenario purpose of the table to which the field belongs; and re-labeling the data category label and information level label corresponding to the field according to the scenario purpose of the table to which the field belongs, thereby obtaining the optimized dataset.
[0088] According to an embodiment of this application, the verification module includes: a matching submodule, used to match the field information to be detected with the rules in the benchmark rule base; a first result module, used to use the classification result determined by the benchmark rule base as the final classification result of the field if the matching is successful; and a second result module, used to use the classification result output by the pre-trained data classification model as the final classification result of the field if the matching is unsuccessful.
[0089] According to an embodiment of this application, the apparatus includes: an instruction receiving module for receiving a user's correction instruction for the final classification result; an adding module for generating new labeled data based on the correction instruction and adding the new labeled data to the initial database to obtain an updated initial database; and an iterative training module for iteratively training the data classification model using the updated initial database.
[0090] According to an embodiment of this application, the device further includes: an output module, configured to output the confidence level corresponding to the classification result; a verification module, further configured to match the field information to be detected with rules in a preset benchmark rule base if the confidence level is higher than a first threshold; a marking and review module, configured to mark the classification result with a first mark and push it to a manual review interface if the confidence level is lower than the first threshold but higher than a second threshold, wherein the first mark indicates that the review status of the classification result is pending review; and a marking and push module, configured to mark the field to be detected with a second mark and push it to the manual review interface if the confidence level is lower than the second threshold, wherein the second mark indicates that the classification result of the field to be detected cannot be determined.
[0091] According to embodiments of this application, any multiple modules among the acquisition module 310, feature extraction module 320, input module 330, and allocation module 340 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the acquisition module 310, feature extraction module 320, input module 330, and allocation module 340 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 310, feature extraction module 320, input module 330, and allocation module 340 may be implemented at least partially as a computer program module that can perform corresponding functions when the computer program module is run.
[0092] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a field-based data management method according to an embodiment of this application.
[0093] like Figure 4 As shown, an electronic device according to an embodiment of this application includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.
[0094] RAM 403 stores various programs and data required for the operation of the electronic device. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 402 and / or RAM 403. It should be noted that programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.
[0095] According to embodiments of this application, the electronic device may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.
[0096] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.
[0097] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 402 and / or RAM 403 and / or one or more memories other than ROM 402 and RAM 403 described above.
[0098] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the field-based data management method provided in the embodiments of this application.
[0099] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0100] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.
[0101] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0102] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0104] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.
Claims
1. A data management method based on field classification, characterized in that, The method includes: Obtain the database table structure field information to be processed; Feature extraction is performed on the field information to obtain numerical features that characterize the field semantics and field context information; The numerical features are input into a pre-trained data classification model to obtain the classification results of the data category and information level corresponding to the field information. The pre-trained data classification model generates an initial data classification model through a pre-constructed initial database, generates an optimized dataset based on the business scenario context information of the database table, and optimizes and trains the initial data classification model based on the optimized dataset. Based on the classification results of the data categories and information levels, fields of different classification results are assigned to different storage partitions of the database.
2. The method according to claim 1, characterized in that, The field information includes field name, data type, table name, and table purpose description. The feature extraction of the field information to obtain numerical features representing field semantics and contextual information includes: The field names are segmented and vectorized to generate semantic feature vectors for the field names; Natural language processing is performed on the table name and table purpose description to generate a table context semantic feature vector; The data type is converted into a numerical feature to obtain the data type numerical feature; The numerical feature is obtained by fusing the semantic feature vector of the field name, the semantic feature vector of the table context, and the numerical feature of the data type.
3. The method according to claim 1, characterized in that, The method further includes: The first batch of labeled data is obtained from historical manual annotation records. The first batch of labeled data consists of field information that has been manually annotated and the corresponding data category labels and information level labels for the field information. Extract the second batch of labeled data from the preset data processing standard table. The second batch of labeled data includes standard field information and standard data category labels and information level labels corresponding to the standard field information. The first batch of labeled data and the second batch of labeled data are merged to form the initial database.
4. The method according to claim 1 or 2, characterized in that, The optimized dataset generated based on business scenario context information from database tables includes: Retrieve fields whose semantics or information level changes under a specified business scenario; Determine the context and purpose of the table to which the field belongs; Based on the scenario and purpose of the table to which the field belongs, the data category label and information level label corresponding to the field are re-labeled to obtain the optimized dataset.
5. The method according to claim 1, characterized in that, After inputting the numerical features into the pre-trained data classification model to obtain the classification results of the data category and information level corresponding to the field information, the process further includes: The field information to be detected is matched with the rules in the preset benchmark rule base; If a match is successful, the classification result determined by the benchmark rule base is used as the final classification result for the field. If the matching fails, the classification result output by the pre-trained data classification model will be used as the final classification result for the field.
6. The method according to claim 1, characterized in that, The method includes: Receive correction instructions from the user regarding the classification results; New labeled data is generated based on the correction instructions, and the new labeled data is added to the initial database to obtain the updated initial database; The data classification model is iteratively trained using the updated initial database.
7. The method according to claim 5, characterized in that, After inputting the numerical features into the pre-trained data classification model to obtain the classification results of the data category and information level corresponding to the field information, the process further includes: Output the confidence level corresponding to the classification result; If the confidence level is higher than the first threshold, then the operation of matching the field information to be detected with the rules in the preset benchmark rule base is performed; If the confidence level is lower than the first threshold and higher than the second threshold, the classification result is marked with a first mark and pushed to the manual review interface. The first mark indicates that the review status of the classification result is pending review. If the confidence level is lower than the second threshold, the field to be detected is marked with a second label and pushed to the manual review interface. The second label indicates that the classification result of the field to be detected cannot be determined.
8. A data management device based on field classification, characterized in that, The device includes: The acquisition module is used to obtain the database table structure field information to be processed; The feature extraction module is used to extract features from the field information to be detected, and obtain numerical features that represent the semantics and context information of the field. The input module is used to input the numerical features into the pre-trained data classification model to obtain the classification results of the data category and information level corresponding to the field information. The pre-trained data classification model generates an initial data classification model through a pre-constructed initial database, generates an optimized dataset based on the business scenario context information of the database table, and optimizes and trains the initial data classification model based on the optimized dataset. The allocation module is used to allocate fields of different classification results to different storage partitions of the database based on the classification results of the data category and information level.
9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.
11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.