Bank data resource checking method and device, computer equipment and storage medium
By preprocessing bank data resources and applying large-scale model technology, the problem of low learning ability of data resource inventory in the existing technology is solved, more efficient and accurate data resource inventory is achieved, and intelligence of data resource management and utilization is promoted. and automation development.
Patent Information
- Application Number
- CN202411533292.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-02-11
AI Technical Summary
The existing data resource inventory method has low learning ability and it is difficult to accurately and efficiently complete the data resource inventory task.
By obtaining data resources and preprocessing, we create training samples, use large models to predict training samples in categories, and map the prediction results to the data resource directory.
It improves the accuracy and efficiency of data resource inventory results, simplifies the process of data resource inventory, and reduces the need for manual intervention.
Smart Images
Figure CN119416055A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of artificial intelligence and financial technology, and in particular to a bank data resource inventory method, device, computer equipment and storage medium. Background Art
[0002] As an important management tool for data resources in the banking industry, the data resource catalog can achieve effective management of data resources and provide multi-dimensional support for algorithm mining, data research and development, business applications, etc. However, traditional data resource inventory methods often rely on manual operations, which is not only time-consuming and labor-intensive, but also prone to errors. With the continuous growth of banking business volume, the number and types of data resources are also increasing, making it increasingly difficult for traditional inventory methods to meet actual needs.
[0003] In order to solve this problem, the industry has begun to explore automated and intelligent data resource inventory methods. Common intelligent data resource inventory methods include those based on machine learning and deep learning. However, machine learning cannot fully understand unstructured text information, and deep learning has limited ability to understand natural language. In other words, inventory methods based on machine learning and deep learning have the problem of low learning ability for data resources, making it difficult to accurately and efficiently complete the task of data resource inventory. Summary of the invention
[0004] The present invention provides a bank data resource inventory method, device, computer equipment and storage medium to solve the technical problem of low learning ability of the existing data resource inventory method.
[0005] First, a method for inventorying bank data resources is provided, including:
[0006] Acquire data resources and perform data resource preprocessing to obtain sample data;
[0007] Creating a training sample based on the sample data;
[0008] Performing category prediction on the training samples based on the large model to obtain a prediction result;
[0009] Evaluate the prediction result, and output a final data item according to the prediction result;
[0010] The final data item is mapped to a data item corresponding to the data resource directory.
[0011] Second, a bank data resource inventory device is provided, including:
[0012] A data acquisition module is used to acquire data resources and perform data resource preprocessing to obtain sample data;
[0013] A sample creation module, used to create training samples according to the sample data;
[0014] A model training module, used to perform category prediction on the training samples based on the large model to obtain a prediction result;
[0015] A training evaluation module, used for evaluating the prediction results and outputting final data items according to the prediction results;
[0016] The data mapping module is used to map the final data item to the data item corresponding to the data resource directory.
[0017] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned bank data resource inventory method when executing the computer program.
[0018] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned bank data resource inventory method are implemented.
[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention ensures the quality of training samples by preprocessing data resources, which helps to reduce the number of large model predictions and improve the efficiency of inventory tasks; and then adopts large model technology to predict the categories of training samples, which can overcome the limitations of traditional machine learning and deep learning in insufficient understanding of text information, achieve more accurate learning and prediction of training samples, and effectively improve the accuracy of data resource inventory results; by mapping data items to data resource directories, the automatic connection of data items is achieved, the process of data resource inventory is simplified, and the need for manual intervention is reduced. The bank data resource inventory method of the present invention provides strong support for the effective management and efficient use of bank data resources, and promotes the development of bank data resource inventory in a more intelligent and automated direction.
[0020] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail as follows. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 This is a schematic diagram of an application environment of a bank data resource inventory method in one embodiment of the present invention;
[0022] Figure 2 It is a flow chart of a method for counting bank data resources in one embodiment of the present invention;
[0023] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S10;
[0024] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S20;
[0025] Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S30;
[0026] Figure 6 yes Figure 2 A schematic flow chart of a specific implementation of step S40;
[0027] Figure 7 yes Figure 2 A schematic flow chart of a specific implementation of step S50;
[0028] Figure 8 It is a structural diagram of a bank data resource inventory device in one embodiment of the present invention;
[0029] Fig. 9 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;
[0030] Fig.10 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail in conjunction with the accompanying drawings and specific implementation methods. The technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0032] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0033] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.
[0034] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0035] See also Figure 1 and Figure 2 , Figure 1 The application scenario diagram of the bank data resource inventory method provided by the embodiment of the present invention. The server can store and obtain data resources, such as data resources of various business systems of the bank, such as Figure 1 As shown, server A and server B store the business data of business platform A and business platform B, and server B obtains the data resources required for inventory from server A and server B, and preprocesses the data resources to obtain sample data; creates training samples based on the sample data; then performs category prediction on the training samples based on the large model to obtain prediction results; evaluates the prediction results, and outputs the final data items based on the prediction results; and then maps the final data items to the data items corresponding to the data resource directory. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.
[0036] See also Figure 2 As shown, Figure 2 A schematic flow chart of a bank data resource inventory method provided by an embodiment of the present invention. The bank data resource inventory method comprises the following steps:
[0037] S10: Acquire data resources and perform data resource preprocessing to obtain sample data.
[0038] The bank data resource inventory method of this embodiment ensures the quality of sample data by analyzing and processing data resources so as to create training samples.
[0039] In some embodiments of the present invention, Figure 3 As shown, a specific data resource preprocessing solution is provided. In S10, data resources are acquired and preprocessed to obtain sample data, which specifically includes the following steps S11-S12.
[0040] S11: Collect data resources within the industry, where the data resources include tables and fields.
[0041] According to the preset format of the data resource directory, the required data resources are obtained from the business, information and other data of various business systems in the industry.
[0042] For example, from the deposit and loan business data:
[0043] Deposit account information: including basic customer information, deposit type, deposit balance, etc.
[0044] Loan contract information: covers loan amount, loan term, interest rate, repayment method, etc.
[0045] Transaction flow: records every transaction made by the customer in the bank, including deposits, withdrawals, transfers, etc.
[0046] Overdue records: Displays detailed information about customers who have not paid on time.
[0047] For example, from personal customer information data:
[0048] Identity information: including name, ID number, contact information, etc.
[0049] Financial information: such as income, expenses, assets, etc.
[0050] Family information: family members, marital status, etc.
[0051] Educational background: academic qualifications, graduation school, etc.
[0052] Among the above data resources, deposit account information, basic customer information, loan contract information, transaction flow, overdue records, identity information, financial information, family information, and educational background can all be presented in the form of tables, and deposit type, deposit balance, loan amount, loan term, interest rate, repayment method, name, ID number, contact information, etc. are fields in the corresponding table.
[0053] S12: Extract, clean and organize the metadata information of the table and field to obtain sample data.
[0054] Extracting metadata of tables and fields facilitates understanding of table and field information and ensures the convenience and accuracy of subsequent data annotation. Metadata provides information on the structure, content, relationship, and management of tables / fields.
[0055] For a table, metadata typically includes:
[0056] Table name: used to uniquely identify the table in the database.
[0057] Table description: A brief description of the table's purpose, content, or business significance.
[0058] Table creation and modification time: records the timestamps of when a table was created and last modified.
[0059] The structure of the table: includes the list of fields in the table, the data type, length, precision, etc. of each field.
[0060] Primary and foreign key information: describes the field or combination of fields that uniquely identifies each row in the table and relates to other tables.
[0061] Index Information: Records the indexes defined on the table to speed up data retrieval.
[0062] Storage and physical properties of the table: such as the storage location, table space, partition information, etc.
[0063] For fields, metadata typically includes:
[0064] Field name: used to uniquely identify the field in the table.
[0065] The data type of the field: such as integer, floating point number, string, date, etc., defines the type of data that the field can store.
[0066] Length and precision of the field: For some data types (such as strings, numbers), you need to specify the length and precision.
[0067] Allow Empty: Indicates whether the field must contain a value.
[0068] Data cleaning of tables and fields is used to remove errors, incompleteness, inaccuracy, duplication, redundancy and other problems in the data to improve data quality and accuracy.
[0069] Data organization of tables and fields is used to normalize data and associate, merge or connect data to facilitate analysis of table and field data.
[0070] S20: Creating training samples based on the sample data.
[0071] After processing the data resources, the sample data is labeled to facilitate big data understanding and learning of the labeled data, which helps to improve the accuracy of model training.
[0072] In some embodiments of the present invention, Figure 4 As shown, a specific solution for creating training samples is provided, in S20, that is, creating training samples according to the sample data, specifically including the following steps S21-S23.
[0073] S21: Establish labeling rules.
[0074] The labeling rules of this embodiment are designed based on expert rules and experience pools. The accuracy of labeling is improved through expert rules and experience pools, thereby optimizing the training effect of the large model.
[0075] Among them, expert rules are guiding principles or standards formulated by industry experts based on their professional knowledge, experience, and in-depth understanding of data annotation, which are highly authoritative and targeted. The experience pool is a knowledge base that stores a large amount of historical annotation data. By searching and comparing the experience pool, it is possible to match which data item some metadata information that is not easy to identify belongs to.
[0076] By formulating annotation rules through expert rules and experience pools, we can ensure that the annotation process not only follows industry best practices, but also makes full use of the effective information in historical annotation references, making the annotation results more accurate and closer to actual application scenarios.
[0077] S22: labeling the sample data according to the labeling rule to obtain a labeling result.
[0078] S23: Perform similarity detection on the annotation results based on the similarity algorithm and expert rules, set a similarity threshold, update the annotation rules using the annotation results with similarity higher than the similarity threshold, and output the sample data with similarity lower than the similarity threshold as training samples.
[0079] For steps S22-S23, the same text data item may not be labeled the same after being labeled by different labeling rules at different time points. For example, when labeling a transaction data, the experience pool labels the transaction data as "transfer", while the expert rule labels the transaction data as "withholding and payment". The similarity algorithm is used to verify the labeling results. If a labeling result with a similarity lower than the similarity threshold is found, it can be determined that the labeling result is inconsistent with the labeling of the existing labeling rule, and the corresponding data sample is used as a training sample, which is helpful for using the large model for accurate text learning and classification, and also helps to train the large model to improve the accuracy of the large model.
[0080] Specifically, in this embodiment, the BLEU (Bilingual Evaluation Understudy) similarity algorithm is used to perform similarity detection on the annotation results. The BLEU similarity algorithm is used to evaluate the quality of machine translation. In the data resource directory inventory method, the algorithm can also be used to verify the consistency and accuracy of the annotation results. The BLEU algorithm is a statistical method based on a large-scale corpus. It judges the quality of text generation by comparing the degree of n-gram overlap between the annotation results and the expert rules. The higher the BLEU score, the more similar the annotation results are to the expert rules, and the better the quality. In addition, the BLEU algorithm cooperates with the large model, which can continuously reduce the similarity threshold on the basis of repeated training and verification of the training samples through the large model technology, so as to reduce the number of training samples and improve the efficiency of model training.
[0081] In specific implementation, after the BLEU algorithm performs similarity detection on the annotation results, a regular expression can also be used to verify whether the format of the annotation results is complete, so as to output complete sample data.
[0082] S30: Perform category prediction on the training samples based on the large model to obtain a prediction result.
[0083] A big model refers to a deep learning model in the field of machine learning and artificial intelligence that is built from a deep neural network, has tens of millions or even hundreds of millions of parameters, and can handle large-scale data and complex tasks. A big model has strong learning and generalization capabilities. The big model trained on a large data set can capture the complex features and potential laws in the data, thereby showing higher accuracy and robustness in the prediction process. Therefore, the use of big model technology to predict the category of training samples can overcome the limitations of traditional machine learning and deep learning in understanding text information, achieve more accurate learning and prediction of training samples, and effectively improve the accuracy of data resource inventory results. With its powerful reasoning and generation capabilities, the big model can shorten the prediction time of training sample categories, improve the overall data processing efficiency, and provide more reliable and efficient support for data resource catalog inventory.
[0084] Category prediction for training samples helps to assign correct category labels to data items when building a data resource directory. Accurate category labels can ensure the accuracy and consistency of the data resource directory and facilitate subsequent data management and use.
[0085] In some embodiments of the present invention, Figure 5 As shown, a specific solution for class prediction of training samples is provided. In S30, class prediction is performed on the training samples based on the large model to obtain a prediction result, which specifically includes the following steps S31-S38.
[0086] S31: Structuring the training samples to obtain structured data items.
[0087] The training samples contain a lot of text data. Structuring the training samples can convert unstructured data into structured data items with clear fields and formats. This not only facilitates subsequent data processing and analysis, but also improves the efficiency of large models in understanding and utilizing data, thereby ensuring the accuracy and reliability of category predictions.
[0088] S32: Building a training model through the ERNIE big model to identify key information of the structured data items.
[0089] Among them, ERNIE (Enhanced Representation through kNowledge IntEgration) is a pre-trained language model. It is a further optimization of the classic large model Bert, which improves the learning and understanding ability of Chinese text. The large model greatly improves the understanding ability of natural language by pre-training on large-scale unlabeled text, and can learn data resources faster and more accurately, that is, metadata information of tables and fields. The ERNIE large model also has the following features:
[0090] Multi-granularity knowledge modeling: ERNIE focuses not only on the representation of words, but also on the representation of phrases, entities, and sentences. This multi-granularity knowledge modeling helps capture the rich structure in language.
[0091] Continuous semantic integration strategy: ERNIE integrates entity recognition and semantic relationship mining in the pre-training stage to better understand the semantics of sentences.
[0092] Structured knowledge fusion: ERNIE uses a large amount of knowledge graph data to integrate structured knowledge into the model during the pre-training stage, improving the model's knowledge representation capabilities.
[0093] Therefore, the use of the ERNIE large model can give full play to ERNIE's advantages in natural language processing and knowledge representation, and efficiently and accurately identify key information in structured data items.
[0094] S33: Based on the DPCNN text classification technology and the key information, text learning and classification are performed on the structured data items to obtain the target category.
[0095] DPCNN, the full name of which is Deep Pyramid Convolutional Neural Networks for Text Categorization, is a deep convolutional neural network. Specifically, it is a text classification technology based on deep learning proposed by Tencent AI-lab. In this solution, it is mainly used to fine-tune ERNIE in the intelligent inventory task.
[0096] The ERNIE large model combined with DPCNN text classification technology can deeply understand and classify structured data items. For example, "Zhang San" is classified as "customer name" so that "Zhang San" can be mapped to "customer name" in the data resource directory.
[0097] S34: performing prediction probability evaluation on the structured data item and the target category to obtain the prediction probability of the structured data item corresponding to the category, and outputting a plurality of structured data items and the corresponding prediction probabilities as prediction results according to the target category set.
[0098] The prediction probability refers to the possibility that a structured data item belongs to the corresponding target category. The prediction probability provides a quantitative evaluation standard and is usually expressed as a value between 0 and 1. The larger the value, the higher the possibility that the structured data item belongs to the target category.
[0099] In the process of text analysis and classification of structured data items, the ERNIE large model and DPCNN text classification technology can preset the success rate of classification to the target category. This step is a bridge connecting model prediction and practical application, which makes the output of the model more practical and valuable for reference.
[0100] S35: Regularly calculate the precision and recall of the training model.
[0101] S36: Calculate the harmonic mean of the precision and the recall according to the precision and the recall.
[0102] S37: Determine whether the training model needs to be optimized based on the harmonic mean.
[0103] S38: Preset a score threshold, compare the harmonic mean with the score threshold, and if the harmonic mean is lower than the score threshold, optimize the training model.
[0104] For steps S35-S38, the performance of the training model is affected by many factors such as the number of training samples, usage time, structural changes, etc. In order to ensure the training efficiency and output accuracy of the training model, the performance of the training model is monitored to optimize the model so that it adapts to new training samples and structural changes. The optimization methods of the model include but are not limited to: adjusting the parameters and structure of the model to improve its generalization ability and accuracy, and using more training samples to train the model to improve its performance.
[0105] Among them, precision, recall and the harmonic mean of precision and recall are important indicators that reflect the performance of the training model.
[0106] Precision: The ratio of samples predicted by the model to samples that are actually positive, which reflects the accuracy of the model's prediction of positive samples. The calculation formula is:
[0107]
[0108] in:
[0109] TP (True Positive): True positive, the number of instances correctly predicted by the model as positive samples.
[0110] FP (False Positive): False positive examples, the number of negative sample instances that the model incorrectly predicts as positive samples.
[0111] Recall rate: The proportion of samples correctly predicted as positive by the model among all true positive samples, which reflects the ability of the model to find all positive samples. The calculation formula is:
[0112]
[0113] in:
[0114] FN (False Negative): False negative examples, the number of positive sample instances that the model incorrectly predicts as negative samples.
[0115] The harmonic mean of precision and recall: used to balance the weight between precision and recall. The higher the F1 score, the better the model performs in both precision and recall. The calculation formula is:
[0116] textF1Score=2×Precision+RecallPrecision×Recall.
[0117] S40: Evaluate the prediction result, and output a final data item according to the prediction result.
[0118] After the training model outputs the structured data items and the corresponding prediction probabilities, a comprehensive and detailed evaluation is conducted on the prediction results to ensure the correctness of the classification of the structured data items, so as to screen out high-quality data items and provide a solid foundation for building an accurate and reliable data resource directory.
[0119] Structured data items are often not isolated, and there is a certain correlation or dependency between structured data items. Therefore, when evaluating the prediction result of any structured data item, this embodiment takes into account the correlation between structured data items, and evaluates the prediction result of the structured data item based on the distribution pattern of all training samples in the prediction results of the training model, so as to further improve the accuracy of the evaluation.
[0120] In some embodiments of the present invention, Figure 6 As shown, a specific solution for evaluating the prediction result is provided, in S40, that is, evaluating the prediction result, and outputting the final data item according to the prediction result, which specifically includes the following steps S41-S44.
[0121] S41: Obtain prediction results under the specified target category.
[0122] S42: Arrange the prediction results in descending order according to the prediction probabilities, and associate the prediction probabilities with the corresponding structured data items.
[0123] For steps S41-S42, determine the target category, such as "personal customer name", and then display the structured data items "Zhang San", "ZHANG SAN", "Li Si", etc. in descending order of predicted probability, so that the structured data item with the highest predicted probability will be arranged in front, and the structured data item with the lowest predicted probability will be arranged in the back.
[0124] S43: Select the predicted probability of the structured data item of the yth item of the target category corresponding to the specified structured data item as the predicted probability threshold of the specified structured data item; wherein y is a times the total number of structured data items corresponding to the target category, and a is greater than 0 and less than 1.
[0125] S44: Outputting the structured data item whose prediction probability exceeds the prediction probability threshold as a final data item.
[0126] For steps S43-S44, in the early stage of intelligent inventory, since the model has limited data items learned, it is necessary to set a prediction probability threshold to filter out the correct attachment results. Traditionally, the prediction results are sampled and judged manually one by one, and the threshold is determined based on the precision and recall rate. Although this method can guarantee a certain accuracy rate, it is inefficient. Therefore, this embodiment sets a prediction probability threshold to automatically evaluate the prediction results, which reduces the consumption of human resources and greatly improves the evaluation efficiency.
[0127] In this embodiment, a is selected as 0.9, that is, the yth structured data item is selected to be at a relatively later position in the sorting, which can not only ensure the pass rate of the structured data item prediction results, but also identify the structured data with extremely low prediction probability so as to reprocess it, thereby improving the accuracy of the data inventory.
[0128] For example, there are 1,000 prediction results for structured data items under the target category "Personal Customer Name". We set the threshold parameter to 0.9, then y = 1000 × 0.9 = 900. Therefore, we select the prediction probability corresponding to the 900th structured data item after sorting as the prediction probability threshold for the "Personal Customer Name" category. If the prediction probability corresponding to the 900th structured data item is 70%, then the prediction probability threshold of the structured data items under this directory category is 70%. If the prediction probability of a structured data item is greater than 70%, it can be output as the final data item. If the prediction probability of a structured data item is less than 70%, it will be re-labeled and trained.
[0129] S50: Map the final data item to a data item corresponding to the data resource directory.
[0130] After training the model and predicting the probability evaluation, the final output items and the corresponding target categories can achieve a high degree of accuracy. By mapping the final data items to the data resource directory according to the target categories, the automatic connection of data items is achieved, which simplifies the process of data resource inventory and reduces the need for manual intervention.
[0131] In some embodiments of the present invention, Figure 7 As shown, a specific final data item mapping solution is provided. In S50, the final data item is mapped to the data item corresponding to the data resource directory, which specifically includes the following steps S51-S52.
[0132] S51: Create a top-down deductive data resource catalog.
[0133] The data resource directory starts from the top level and is refined layer by layer in the form of subject-directory-data item until all the data resources of the bank are covered. This top-down deductive approach makes the data resource directory structure clear, easy to manage and query, and can fully cover the bank's data resources to avoid omissions, which helps to ensure the integrity and systematicness of the data resource directory.
[0134] The following is an example implementation of a data resource directory:
[0135] The subject is: Client;
[0136] The categories under the customer theme are: general information, individual customers, and corporate customers;
[0137] The data items in the general information are: customer number, name, contact information, and registration date.
[0138] S52: The final data item is mounted on the data resource directory from bottom to top.
[0139] The target categories corresponding to the final data items are attached to the data resource directory from bottom to top, that is, first determine the data items corresponding to the data resource directory to which the final data items are attached, and then gradually classify them into higher-level directories and topics. This method ensures that each final data item can be accurately classified and mapped, avoiding data confusion or incorrect mapping; and the bottom-up mapping method allows for flexible adjustment and modification at the data item level. With the development of banking business and the emergence of new data resources, it may be necessary to add or modify data items. In this case, the bottom-up mapping method can more easily adapt to these changes without the need for large-scale reconstruction of the entire directory structure.
[0140] It can be seen that in the above scheme, by preprocessing the data resources and ensuring the quality of the training samples, it is helpful to reduce the number of large model predictions and improve the efficiency of the inventory task; then the large model technology is used to predict the categories of the training samples, which can overcome the limitations of traditional machine learning and deep learning in understanding text information, achieve more accurate learning and prediction of training samples, and effectively improve the accuracy of the data resource inventory results; by mapping the data items to the data resource directory, the automatic connection of the data items is achieved, the process of data resource inventory is simplified, and the need for manual intervention is reduced. The bank data resource inventory method of the present invention provides strong support for the effective management and efficient use of bank data resources, and promotes the development of bank data resource inventory in a more intelligent and automated direction.
[0141] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0142] In one embodiment, the present invention provides a bank data resource inventory device, which corresponds one-to-one to the bank data resource inventory method in the above embodiment. Figure 8 As shown, the bank data resource inventory device includes a data acquisition module 101, a sample creation module 102, a model training module 103, a training evaluation module 104 and a data mapping module 105. The detailed description of each functional module is as follows:
[0143] The data acquisition module 101 is used to acquire data resources and perform data resource preprocessing to obtain sample data;
[0144] A sample creation module 102, used to create a training sample according to the sample data;
[0145] A model training module 103 is used to perform category prediction on the training samples based on the large model to obtain a prediction result;
[0146] A training evaluation module 104, used to evaluate the prediction results and output final data items according to the prediction results;
[0147] The data mapping module 105 is used to map the final data item to a data item corresponding to the data resource directory.
[0148] In one embodiment, the data acquisition module 101 is specifically used for:
[0149] Collect data resources within the industry, the data resources including tables and fields;
[0150] The metadata information of the table and fields is extracted, cleaned and sorted to obtain sample data.
[0151] In one embodiment, the sample creation module 102 is specifically used to:
[0152] Establish labeling rules;
[0153] Annotating the sample data according to the annotation rule to obtain an annotation result;
[0154] Based on the similarity algorithm and expert rules, similarity detection is performed on the annotation results, a similarity threshold is set, the annotation rules are updated using the annotation results with similarity higher than the similarity threshold, and the sample data with similarity lower than the similarity threshold is output as training samples.
[0155] In one embodiment, the model training module 103 is specifically used to:
[0156] Structuring the training samples to obtain structured data items;
[0157] Building a training model through the ERNIE big model to identify key information of the structured data items;
[0158] Based on the DPCNN text classification technology and the key information, text learning and classification are performed on the structured data items to obtain a target category;
[0159] The structured data items and the target categories are evaluated for prediction probability to obtain the prediction probability of the structured data items corresponding to the categories, and a number of structured data items and corresponding prediction probabilities are output as prediction results according to the target category set.
[0160] Regularly calculating the precision and recall of the training model;
[0161] According to the precision and recall, calculate the harmonic mean of the precision and the recall;
[0162] Determining whether the training model needs to be optimized according to the harmonic mean;
[0163] A score threshold is preset, and the harmonic mean is compared with the score threshold. If the harmonic mean is lower than the score threshold, the training model is optimized.
[0164] In one embodiment, the training evaluation module 104 is specifically configured to:
[0165] Get the prediction results for the specified target category;
[0166] Arrange the prediction results in descending order according to the prediction probabilities, and associate the prediction probabilities with the corresponding structured data items;
[0167] Select the predicted probability of the structured data item of the yth item of the target category corresponding to the specified structured data item as the predicted probability threshold of the specified structured data item; where y is a times the total number of structured data items corresponding to the target category, and a is greater than 0 and less than 1;
[0168] The structured data item whose prediction probability exceeds the prediction probability threshold is output as a final data item.
[0169] In one embodiment, the data mapping module 105 is specifically used to:
[0170] Create a top-down deductive data resource catalog;
[0171] The final data item is mounted on the data resource directory from bottom to top.
[0172] The specific definition of the bank data resource inventory device can be found in the definition of the bank data resource inventory method above, which will not be repeated here. Each module in the above-mentioned bank data resource inventory device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0173] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a service-side method for inventorying bank data resources.
[0174] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a client side of a bank data resource inventory method.
[0175] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:
[0176] Acquire data resources and perform data resource preprocessing to obtain sample data;
[0177] Creating a training sample based on the sample data;
[0178] Performing category prediction on the training samples based on the large model to obtain a prediction result;
[0179] Evaluate the prediction result, and output a final data item according to the prediction result;
[0180] The final data item is mapped to a data item corresponding to the data resource directory.
[0181] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.
[0182] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0183] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0184] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A bank data resource inventory method, characterized in that: include: Acquire data resources and perform data resource preprocessing to obtain sample data; Creating a training sample based on the sample data; Performing category prediction on the training samples based on the large model to obtain a prediction result; Evaluate the prediction result, and output a final data item according to the prediction result; The final data item is mapped to a data item corresponding to the data resource directory.
2. The bank data resource inventory method according to claim 1, characterized in that: The acquiring of data resources and preprocessing of the data resources to obtain sample data includes: Collect data resources within the industry, the data resources including tables and fields; The metadata information of the table and fields is extracted, cleaned and sorted to obtain sample data.
3. The bank data resource inventory method according to claim 2, characterized in that: The step of creating a training sample according to the sample data includes: Establish labeling rules; Annotating the sample data according to the annotation rule to obtain an annotation result; Based on the similarity algorithm and expert rules, similarity detection is performed on the annotation results, a similarity threshold is set, the annotation rules are updated using the annotation results with similarity higher than the similarity threshold, and the sample data with similarity lower than the similarity threshold is output as training samples.
4. The bank data resource inventory method according to claim 3, characterized in that: The performing category prediction on the training sample based on the large model to obtain a prediction result includes: Structuring the training samples to obtain structured data items; Building a training model through the ERNIE big model to identify key information of the structured data items; Based on the DPCNN text classification technology and the key information, text learning and classification are performed on the structured data items to obtain a target category; The structured data items and the target categories are evaluated for prediction probability to obtain the prediction probability of the structured data items corresponding to the categories, and a number of structured data items and corresponding prediction probabilities are output as prediction results according to the target category set.
5. The bank data resource inventory method according to claim 4, characterized in that: After the text learning and classification of the structured data items by using the ERNIE large model and DPCNN text classification technology to obtain the target category, the method further includes: Regularly calculating the precision and recall of the training model; According to the precision and recall, calculate the harmonic mean of the precision and the recall; Determining whether the training model needs to be optimized according to the harmonic mean; A score threshold is preset, and the harmonic mean is compared with the score threshold. If the harmonic mean is lower than the score threshold, the training model is optimized.
6. The bank data resource inventory method according to claim 5, characterized in that: The step of evaluating the prediction result and outputting a final data item according to the prediction result includes: Get the prediction results for the specified target category; Arrange the prediction results in descending order according to the prediction probabilities, and associate the prediction probabilities with the corresponding structured data items; Select the predicted probability of the structured data item of the yth item of the target category corresponding to the specified structured data item as the predicted probability threshold of the specified structured data item; where y is a times the total number of structured data items corresponding to the target category, and a is greater than 0 and less than 1; The structured data item whose prediction probability exceeds the prediction probability threshold is output as a final data item.
7. The method for counting bank data resources according to claim 6, characterized in that: Mapping the final data item to a data item corresponding to the data resource directory includes: Create a top-down deductive data resource catalog; The final data item is mounted on the data resource directory from bottom to top.
8. A bank data resource inventory device, characterized in that: include: A data acquisition module is used to acquire data resources and perform data resource preprocessing to obtain sample data; A sample creation module, used to create training samples according to the sample data; A model training module, used to perform category prediction on the training samples based on the large model to obtain a prediction result; A training evaluation module, used for evaluating the prediction results and outputting final data items according to the prediction results; The data mapping module is used to map the final data item to the data item corresponding to the data resource directory.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the bank data resource inventory method as described in any one of claims 1 to 7 are implemented.
10. A storage medium, wherein the computer-readable storage medium stores a computer program, characterized in that: When the computer program is executed by a processor, the steps of the bank data resource inventory method as described in any one of claims 1 to 7 are implemented.