Large model-based data immunity construction method and data immunity method

Abstract

The invention discloses a data immunity construction method based on a large model and a data immunity method, and relates to the technical field of data immunity. The method comprises the steps of obtaining basic data of a target field; determining an immunodetection rule for detecting the problem data based on the problem type of the problem data in the basic data; extracting data features of the basic data, and obtaining a problem data set based on the data features and the immunodetection rule; constructing an immune algorithm rule for repairing problem data based on the problem data set, wherein the immune algorithm rule comprises normalization processing based on a large model; and forming data immunity applied to the target domain based on the immune detection rule and the immune algorithm rule. According to the implementation mode provided by the invention, the data problem can be found and the data problem can be automatically repaired by constructing the data immunity system, so that the understandability, normalization and safety of the data are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data immunity, in particular to a data immunity construction method based on a large model, a data immunity method, a data immunity construction device based on a large model, a computer readable storage medium and a computer program product. BACKGROUND

[0002] With the development of big data technology, the application scenarios of massive data have been greatly expanded, and the application value has been greatly improved. However, there are many problems in the data quality of massive data, which can only be found and processed in application, and have a great impact on application effect. Taking a human resource management system as an example, the data of the existing human resource management system has the shortcomings of large data volume, scattered data, insufficient data value and data specification, which has a great impact on data application.

[0003] The existing system data often has the shortcomings of large data volume, insufficient data value and data specification. For example, in a pile of existing data, if the source and blood relationship of the data are unknown, and the purpose of the data is also unknown, then the existence value and cost of the data are difficult to determine, and the overall certainty of the data is very low. For existing data, it is usually necessary to discover data problems through various methods and then perform post-treatment, that is, to pollute first and then treat. Therefore, from the aspects of data understandability, specification, security and cost, it is necessary to improve the certainty of data and promote the pre-warning and processing of data quality problems. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a data immunity construction method based on a large model and a data immunity method, to construct a data immunity system, to automatically scan data problems in the process of using data, to discover data problems, and to use rules and model methods, so that the data immunity system has the function of automatically repairing data problems, to at least solve part of the technical problems in the background art.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a method, which includes: acquiring basic data of a target field; determining an immune detection rule for detecting problem data based on the problem type of the problem data in the basic data; extracting data features of the basic data, and obtaining a problem data set based on the data features and the immune detection rule; constructing an immune algorithm rule for repairing problem data based on the problem data set, the immune algorithm rule including a normalization processing based on a large model; and forming a data immunity applied to the target field based on the immune detection rule and the immune algorithm rule.

[0006] Optionally, the basic data includes data recorded by a database or a data table.

[0007] Optionally, the problem data includes missing data, duplicate data, error data, and unusable data, and the problem types include data missing, data duplication, data error, and data unusable.

[0008] Optionally, the data features include values and attributes of the data, and the immune detection rules include data constraints, business rule constraints, and data operation constraints; and extracting the data features of the base data and obtaining the problem data set based on the data features and the immune detection rules includes: adding data in an intermediate state detected by a database transaction to the problem data set; adding duplicate operation result data obtained based on a duplicate operation to the problem data set; adding null data obtained based on data check missing to the problem data set; adding format error data based on data entry errors to the problem data set; and adding character error data based on data entry errors to the problem data set.

[0009] Optionally, the immune algorithm rules for repairing the problem data are constructed based on the problem data set, including: determining data operations in the immune algorithm rules according to the problem types of the problem data, the data operations including one of supplement, deletion, format conversion, correction, and matching; and determining operation parameters in the immune algorithm rules according to the immune detection rules corresponding to the problem types, the operation parameters including: a supplemented value, a deletion target, a target format, a corrected value, and a matching result.

[0010] Optionally, the data operations in the immune algorithm rules are determined according to the problem types of the problem data, including: if the problem type is data missing, determining the data operation as supplement; if the problem type is data duplication, determining the data operation as deletion; if the problem type is data error, determining the data operation as format conversion when there is format error, and determining the data operation as correction when there is value error; and if the problem type is data unusable, determining the data operation as matching.

[0011] Optionally, the operation parameters in the immune algorithm rules are determined according to the immune detection rules corresponding to the problem types, including: generating a supplemented value with data format constraints within a numerical range constraint based on the data constraints in the immune detection rules; or determining a deletion target from duplicate data based on business rule constraints in the immune detection rules, and determining a deletion target from duplicate data based on generation time of the duplicate data if the immune detection rules do not include business rule constraints; or generating a target format based on data format constraints in the data constraints of the immune detection rules, and generating a corrected value based on numerical range constraints in the data constraints of the immune detection rules; if the data error includes data inconsistency, determining a corrected value from inconsistent data based on a trained artificial intelligence algorithm; or generating a matching result based on a combined model obtained by combining a large model and a small model.

[0012] Optionally, the trained artificial intelligence algorithm is obtained by the following steps: determining a data dimension of an input of the artificial intelligence algorithm; extracting an entity and an entity relationship between entities related to the data dimension from the basic data; obtaining extracted data based on the data dimension, an entity label of the entity, and an entity relationship type using an extraction model; dividing the extracted data into a training set and a verification set, training the artificial intelligence algorithm using the training set, verifying the trained artificial intelligence algorithm using the verification set, and obtaining the trained artificial intelligence algorithm after verification.

[0013] Optionally, the matching result generated by the combined model obtained based on the combination of the large model and the small model comprises: extracting a keyword of the unavailable data; inputting the keyword into the small model for normalization processing, obtaining a first normalization result if the small model normalization is successful; if the small model normalization fails, inputting the keyword into the large model for normalization processing, obtaining a second normalization result if the large model normalization is successful, the large model and the small model being distinguished by structural complexity and parameter order of magnitude; and performing standardization checking and version conversion on the first normalization result or the second normalization result to obtain the matching result.

[0014] Optionally, the method further comprises: before inputting the small model, searching for the keyword in a mapping table, if there is a search result, taking the search result as the matching result, and no longer inputting the keyword into the small model; if there is no search result, inputting the keyword into the small model for normalization processing; and after performing standardization checking and version conversion on the first normalization result or the second normalization result to obtain the matching result, storing the keyword and the corresponding matching result in the mapping table.

[0015] Optionally, the data immunity applied to the target field comprises the immune detection rule and the immune algorithm rule; the immune detection rule is used to identify problem data from the basic data of the target field; and the immune algorithm rule is used to repair data problems in the problem data.

[0016] Optionally, the target field comprises a human resource field, and the basic data of the target field comprises at least one of employee basic information, organization information, performance information, training and development information, work resume information, labor contract information, salary information, various configuration code information, and employee-related dimension indicators.

[0017] Also provided in the present application is a data immunization method, comprising: obtaining to-be-processed data; inputting the to-be-processed data into a data governance model with data immunity, the data immunity being obtained according to the foregoing data immunity construction method based on a large model; and obtaining data after data governance.

[0018] Also provided in the present application is a data immunity construction device based on a large model, comprising: a data acquisition module configured to acquire basic data of a target field; an immune detection module configured to determine immune detection rules for detecting problem data based on a problem type of the problem data in the basic data; a problem data module configured to extract data features of the basic data, and obtain a problem data set based on the data features and the immune detection rules; an immune repair module configured to construct immune algorithm rules for repairing the problem data based on the problem data set, the immune algorithm rules comprising normalization processing based on a large model; and an immune output module configured to form data immunity applied to the target field based on the immune detection rules and the immune algorithm rules.

[0019] Also provided in the present application is a data immunity device, comprising: a data scanning module configured to acquire to-be-processed data; an immune processing module configured to input the to-be-processed data into a data governance model with data immunity, the data immunity being obtained according to the foregoing data immunity construction method based on a large model; and a data output module configured to obtain data after data governance.

[0020] Also provided in the present application is a computer-readable storage medium, the storage medium storing instructions, when the instructions are executed on a computer, causing the computer to perform the steps of the foregoing data immunity construction method based on a large model.

[0021] Also provided in the present application is a computer program product, comprising a computer program, the computer program, when executed by a processor, implementing the foregoing data immunity construction method based on a large model.

[0022] The foregoing technical solutions have at least the following beneficial effects: internal and external data are fully collected, a data immunity system is constructed, and various data problems are improved; data immunity is innovatively applied to data governance, thereby promoting the automation and intelligentization of data governance.

[0023] Other features and advantages of the embodiments of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings are included to provide a further understanding of embodiments of the application, and are incorporated in and constitute a part of the specification, illustrate embodiments of the application, and together with the description serve to explain the principles of the application. In the drawings: Figure 1 A schematic diagram of steps of a large model-based data immunity construction method according to embodiments of the application is shown schematically; Figure 2 A schematic diagram of an application environment of a large model-based data immunity construction method according to embodiments of the application is shown schematically; Figure 3 A schematic diagram of the principle of a model-based information extraction process according to embodiments of the application is shown schematically; Figure 4 A schematic diagram of the correlation of performance compensation according to embodiments of the application is shown schematically; Figure 5 A schematic diagram of the process of large model arrangement according to embodiments of the application is shown schematically; Figure 6 A schematic diagram of the principle process of a data immunity method according to embodiments of the application is shown schematically; Figure 7 A schematic diagram of the structure of a large model-based data immunity construction device according to embodiments of the application is shown schematically; Figure 8 A schematic diagram of the internal structure of a computer device according to embodiments of the application is shown schematically. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described below in a clear, complete manner with reference to the drawings in the embodiments of the application. It should be understood that the specific embodiments described herein are merely used to explain and illustrate the embodiments of the application, and are not used to limit the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the application.

[0026] It should be noted that if there is a directional indication (such as up, down, left, right, front, back, etc.) in the embodiments of the application, the directional indication is only used to explain the relative positional relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indication also changes accordingly.

[0027] In addition, if the description of "first", "second" and the like is involved in the embodiments of the present application, the description of "first", "second" and the like is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of the ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the protection scope required by the present application.

[0028] The acquisition, storage, use, processing and the like of data in the technical solutions of the present application comply with the relevant provisions of national laws and regulations.

[0029] It should be noted that in the embodiments of the present application, some existing industry solutions of software, components, models and the like may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0030] Figure 1 The steps of the data immunity construction method based on large model according to the embodiments of the present application are schematically shown. As shown in Figure 1 The data immunity construction method based on large model includes: S01, acquiring basic data of a target field; the target field here refers to the field to which the data immunity needs to act, and the data dimensions, rule limitations and association relationships of data in each target field may be different. The basic data mainly refers to the original data recorded or stored, which may be distributed in different data tables of the same database, or in different databases, or on different data servers.

[0031] S02, determining an immune detection rule for detecting problem data based on the problem type of the problem data in the basic data; S03, extracting data features of the basic data, and obtaining a problem data set based on the data features and the immune detection rule; S04, constructing an immune algorithm rule for repairing problem data based on the problem data set, the immune algorithm rule including a normalization processing based on a large model; S05, forming a data immunity applied to the target field based on the immune detection rule and the immune algorithm rule.

[0032] Through the above implementation, by constructing data immunity, data problems can be automatically scanned and discovered in the process of using data, and through analysis and experience accumulation of data knowledge, the understandability, standardization and security of business data are finally fed back to assist, and the data governance cost is also reduced.

[0033] Figure 2 The application environment schematic diagram of the data immunity construction method based on a large model according to the embodiment of the application is schematically shown. As shown in Figure 2 The data immunity construction method based on a large model provided by the application can be applied in the application environment as shown in Figure 2 The terminal 102 communicates with the server 104 through the network. The terminal 102 is a data source end, such as a book system, a database or a database cluster, which can be but is not limited to various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers, which runs the aforementioned data immunity construction method, constructs and stores a program or a functional module with data immunity. Similarly, the data immunity method also runs in the server 104, and the data in the terminal 102 is processed by the server 104 to realize data immunity.

[0034] In some embodiments of the present application, the basic data includes data recorded by a database or a data table. Taking the target field of human resources as an example, the basic data of the target field includes at least one of employee basic information, organization information, performance information, training and development information, work resume information, labor contract information, salary information, various configuration code information, and employee-related dimension indicators. Specifically, the employee basic information includes employee number, name, gender, birth date, age, education, degree, ID number, post, employee status, marital status, family address, professional and technical qualification, organization, region, job title, job level, work experience, marital status, telephone, email, and remarks. The organization information includes organization number, organization name, organization category, organization level, organization layer, organization status, organization photo, business address, business scope, postal code, communication address, telephone number, organization head, organization affiliation code, organization order, organization region level, and remarks. The performance information includes employee number, work performance, skills, advantages, disadvantages, improvement areas, performance unit administrator, and remarks. The training and development information includes employee number, courses studied, course classification, certificates obtained, skills mastered, and field direction. The work resume information includes employee number, unit, work, job title, job level, employee work history information, position, employment date, termination date, and reason for leaving. The labor contract information includes employee number, employee attribute, contract content, contract start date, contract end information, labor information, labor dispute situation, contract term, contract processing category, agreement category, contract number, declassification period, and remarks. The salary information includes employee number, basic salary, post salary, performance salary, other salary, welfare information, individual tax information, and the like. The various configuration code information includes table name, field name, field value, start date, end date, description, and remarks. The above enumeration only illustrates the basic data that may exist in the field of human resources, and in actual scenarios, those skilled in the art can add or subtract based on the needs.

[0035] In some embodiments of the present application, the problem data includes missing data, duplicate data, incorrect data, and unusable data; and the problem types include data missing, data duplication, data incorrect, and data unusable, respectively. Examples of the above problem data are as follows: the first type, missing data, for example, the education of an employee is null, which can be caused by the administrator forgetting to record or being mistakenly deleted. The second type, duplicate data, for example, the address and name of an employee are the same, but if the primary key is not processed, other data are mostly the same, such as the position, rank, training course, etc., which need to be processed according to the actual situation. The third type, incorrect data, for example, the data is not recorded or updated according to the standard, regulation or rule, which can be divided into several cases as follows: (1) data format error, for example, a number format is recorded as a date format; (2) data inconsistency, for example, the name is recorded as: Zhang Sanfeng, Sanfeng, ZHANSANFENG, zhangsanfeng, etc.; (3) abnormal value, for example, the rank of an employee is 1-12, but the problem data appears 0, 13, etc. The fourth type, unusable data, for example, the address prefix is "Xiamen Siming District", and if the data of the "district" level needs to be counted, analyzed or classified, "Siming" needs to be identified. Through the definition of the above problem data, a data basis is provided for constructing the problem data set.

[0036] In some embodiments of the present application, the data features include values and attributes of the data, and the immune detection rules include data constraints, business rule constraints, and data operation constraints; extracting data features of the base data, and obtaining a problem data set based on the data features and the immune detection rules, includes: adding data in an intermediate state detected by a database transaction to the problem data set; adding repeated operation result data obtained based on repeated operations to the problem data set; adding null data obtained based on missing data validation to the problem data set; adding format error data based on data entry errors to the problem data set; and adding character error data based on data entry errors to the problem data set. The immune detection rules are used to detect problem data, and need to be set according to the aforementioned problem types. The present embodiment extracts the causes of problem data generation, and provides multiple ways to obtain a problem data set, including: adding data in an intermediate state detected by a database transaction to the problem data set; the database has multiple data anomaly detection mechanisms built-in, such as statistical analysis, machine learning, rule engine, time series analysis, etc., when data in the database is abnormal, the database transaction will set it to an intermediate state, at this time, the data can be added to the problem data set as problem data. Adding repeated operation result data obtained based on repeated operations to the problem data set, some data exists repeatedly, and the possible reasons include: multiple clicks on the same function, or multiple entries, or multiple insertions, etc. Adding null data obtained based on missing data validation to the problem data set, some non-empty fields store null values, and the possible reasons include: the database dictionary may not be validated and checked, or historical data is not well handled for non-empty migration data. Adding format error data based on data entry errors to the problem data set, for example, a phone number contains Chinese, and the possible reasons include: illegal data is entered during entry, and is not checked. Adding character error data based on data entry errors to the problem data set, for example, extra spaces are stored, and the possible reasons include: incomplete deployment causes some illegal data to be generated. The above approaches illustrate the generation of a problem data set.

[0037] In some embodiments of the present application, the immune algorithm rule for repairing the problem data is constructed based on the problem data set, including: determining the data operation in the immune algorithm rule according to the problem type of the problem data, the data operation including one of supplement, deletion, format conversion, correction and matching; determining the operation parameter in the immune algorithm rule according to the immune detection rule corresponding to the problem type, the operation parameter including: the value of supplement, the deletion target, the target format, the corrected value and the matching result. In the construction of the data immune system, after detecting the problem data, it also needs to be automatically repaired. Automatic repair needs to be set according to the problem type, for example, if the data is repeated, the repeated data needs to be deleted, and if the data is missing, the data needs to be supplemented. Therefore, the immune algorithm rule in the present embodiment is determined step by step through data operation and operation parameter, which is beneficial to realize the flexibility of immune algorithm rule generation.

[0038] In some embodiments of the present application, the data operation in the immune algorithm rule is determined according to the problem type of the problem data, including: if the problem type is data missing, determining the data operation as supplement; if the problem type is data repetition, determining the data operation as deletion; if the problem type is data error, determining the data operation as format conversion when the format is wrong, and determining the data operation as correction when the value is wrong; if the problem type is data unavailable, determining the data operation as matching. The above corresponding mode can adopt the list mapping mode to directly determine the data operation according to the problem type.

[0039] In some embodiments of the present application, the operation parameters in the immune algorithm rule are determined according to the immune detection rule corresponding to the problem type, including: generating a supplementary value with a data format constraint within a numerical range constraint based on the data constraint in the immune detection rule, for missing data, in order not to affect the accuracy of the data analysis and mining result, we can compensate for data missing during data application, or exclude null values from data application, we can selectively fill in some median, average, random number, etc., and label these data for subsequent problem checking and backtracking. After one or more rounds of value filling, if there are still other missing data in the database that are difficult to backfill, the system can be introduced again or entered and calculated according to the business rules. Or based on the business rule constraint in the immune detection rule, determine the deletion target from the duplicate data, if the immune detection rule does not include the business rule constraint, determine the deletion target from the duplicate data based on the generation time of the duplicate data, for duplicate data, only the redundant duplicate data needs to be removed, and the latest data is retained; for duplicate data with business rules, it may need to be processed through business rules or business judgment. Or generate a target format based on the data format constraint in the data constraint of the immune detection rule, generate a corrected value based on the numerical range constraint in the data constraint of the immune detection rule; if the data error includes non-uniform data, determine the corrected value from the non-uniform data based on artificial intelligence algorithm, for error data, for format error data, it needs to be found through the internal logic structure of the system; for outliers, they can be excluded by limiting the interval; for non-uniform data, it may not be possible to solve from the database, we can use artificial intelligence algorithm model to train and judge, make matching algorithm rules, and use algorithms and rules to associate and standardize these data. Or generate a matching result based on a combined model obtained by combining a large model and a small model. For unusable data, many solutions can be obtained through keyword matching method. The present embodiment can provide more accurate immune measures for different data problems and obtain immune results that are more in line with the actual situation of the data.

[0040] In some embodiments of the present application, the trained artificial intelligence algorithm is obtained by the following steps: determining the data dimension of the input of the artificial intelligence algorithm; extracting entities related to the data dimension and entity relationships between entities from the basic data; obtaining extracted data based on the data dimension, entity label of the entity and entity relationship type using an extraction model; dividing the extracted data into a training set and a validation set, training the artificial intelligence algorithm using the training set, and verifying the trained artificial intelligence algorithm using the validation set, to obtain the trained artificial intelligence algorithm after verification. Figure 3A schematic diagram of a principle of implementing an information extraction process based on a model according to an embodiment of the present application is shown schematically. As shown in Figure 3 The extractive model can be provided with multiple models based on different specific functions. After the data in the database is structured and extracted, the personnel information extraction model and the pre-training model are processed, and finally the normalized result is obtained. Among them, the model used for structured extraction and the personnel information extraction model are the extractive model described above.

[0041] Taking the data dimension as the performance salary information as an example, the above embodiments are described as follows: the important information of personnel is contained in the “four categories” of key information, and 14 types of entities and their relationships are extracted from the performance salary. Figure 4 A schematic diagram of the correlation relationship of performance salary according to an embodiment of the present application is shown schematically. As shown in Figure 4 The entity types related to performance salary include: basic salary, post salary, fixed performance salary, floating performance salary, holiday allowance, basic endowment insurance, basic medical insurance, unemployment insurance, work injury insurance, housing accumulation fund payment, annual performance appraisal, performance appraisal level, key performance indicator, promotion situation; relationship types: time, duration, attendance, related, reason (root), salary and appraisal method, salary interval, salary and appraisal result, performance result reason, negative word, etc. Thus, an example of the performance salary information of a certain employee is obtained: the employee's appraisal result is “to be improved” (appraisal level: C), time: 2023, duration: 1 year, floating performance salary: -5567; the performance appraisal results in the past three years are: good B+, competent B, to be improved C; the attendance situation is: 28 times of tardiness in 2023, less than 8 hours 12 times; the performance result reason is: the initial judgment of work attitude, work motivation and work ability has decreased.

[0042] Using the extractive model, input the entity label, entity relationship type and the information to be extracted, and the output result is part of the information segment. The database mode is as follows: {“performance”: [“time”, “duration”, “attendance”, “related”, “reason (root)”, “salary and appraisal method”, “salary and appraisal result”, “performance result reason”, “negative word”] “salary”: [“time”, “duration”, “attendance”, “related”, “reason (root)”, “salary and appraisal method”, “salary interval”, “salary and appraisal result”, “negative word”] “attendance”: [“attendance”, “negative word”] “work attitude”: [“time”, “duration”, “attendance”, “negative word”] “work motivation”: [“salary interval”, “salary and appraisal result”, “performance result reason”, “negative word”] “workability”: [“about”, “reason (root)”, “performance result reason”, “negative word”]}.

[0043] In order to ensure the diversity of data, the personnel information of this round of labeled data is obtained from 5 different companies. A total of 20000 pieces of labeled data are labeled, and a total of 14 entity types and 10 entity relationship types are labeled. The labeled data of different companies are divided into training set, test set and validation set according to the proportions of 0.2, 0.6 and 0.7, which are used for model training, statistical test index and model effect evaluation.

[0044] The model effect evaluation is as follows: precision P = number of predicted correct entities / number of predicted entities; recall R = number of predicted correct entities / number of labeled entities; comprehensive evaluation F1 = 2*P*R / (P+R). The test index results are as follows:

[0045] As can be seen, in the data immunity process, the large model technology is used to establish data immunity, so that the data has standardization and self-regulation in the whole production and use process, and the data self-recovery ability is improved.

[0046] In some embodiments of the present application, the combined model generated based on the combination of large models and small models generates a matching result, including: extracting keywords of unavailable data; inputting the keywords into the small model for normalization processing, if the small model normalization is successful, obtaining a first normalization result; if the small model normalization fails, inputting the keywords into the large model for normalization processing, if the large model normalization is successful, obtaining a second normalization result, the large model and the small model are distinguished by structure complexity and parameter order of magnitude; the first normalization result or the second normalization result is subjected to standardization checking and version conversion to obtain the matching result. This embodiment extracts keywords and uses the combination of large and small models to process. The large model uses a private deployment personnel information vertical large model. The large model usually uses a deeper and wider neural network, such as a deep convolutional neural network (such as ResNet, Inception, EfficientNet) or a large transformer model (such as BERT, GPT) and the like. These models have millions to tens of billions, hundreds of billions of parameters, which can capture more features and complexity. The neural network architecture used by the small model usually includes a shallow neural network, a lightweight convolutional neural network (such as MobileNet) or a small recurrent neural network (such as LSTM or GRU) and the like. The small model may have only a few layers and relatively few parameters to maintain the lightweight characteristics of the model. The present application combines the mixed arrangement of small and large models, which takes into account the processing speed and processing quality of the model.

[0047] In some embodiments of the present application, the method further comprises: before inputting the small model, searching the keyword in a mapping table, if there is a search result, taking the search result as the matching result, and no longer inputting the keyword into the small model; if there is no search result, inputting the keyword into the small model for normalization processing; and after the first normalization result or the second normalization result is standardized and version-converted to obtain the matching result, storing the keyword and the corresponding matching result in the mapping table. Figure 5 A flowchart of the small model arrangement according to the embodiments of the present application is schematically shown. As shown in Figure 5 In the case where the small model and the large model both fail to normalize, the data is written into a non-standardization record library. The non-standardization record library can process data through manual intervention and the like, which can reduce the complexity of program rules and reduce a large number of manual processing operations. In this case, we can record the abnormal situation to form a data analysis report, similar to the treatment of patients by doctors, medical devices and medicines. In this way, personnel can handle the corresponding processing in the later stage according to the report, which can save a part of manpower and improve the data immunity effect. If it has been normalized and saved in the mapping table, the mapping table is searched before each model processing to avoid repeated processing of subsequent models, so as to avoid waste of model computing power.

[0048] In some embodiments of the present application, the data immunity applied to the target field includes the immune detection rule and the immune algorithm rule; the immune detection rule is used to identify problem data from the basic data of the target field; and the immune algorithm rule is used to repair data problems in the problem data. The data immunity constructed in this way can continuously exchange and use data, scan data problems, find data problems, and automatically repair data problems.

[0049] In some embodiments of the present application, the target field includes a human resource field, and the basic data of the target field includes at least one of employee basic information, organization information, performance information, training and development information, work resume information, labor contract information, salary information, various configuration code information, and employee-related dimension indicators. This embodiment is implemented in the target field of the human resource field, and the specific information such as employee basic information is as described above, which is not repeated here.

[0050] As can be seen from the above embodiments, the data immunity constructed by the data immunity construction method based on the large model in the present application has the function of automatically repairing data problems, and through analysis and mining of data knowledge and accumulation of experience, it finally helps to improve the understandability, standardization and security of business data.

[0051] Also provided in the present application is a data immunization method, comprising: obtaining to-be-processed data; inputting the to-be-processed data into a data immunization data governance model, wherein the data immunization is obtained according to the aforementioned data immunization construction method based on a large model; and obtaining data after data governance. The data immunization can identify data with diseases, problems or errors in data records in advance, and then automatically remove these problematic data, which is an automatic data checking process, can remove invalid and repetitive data, eliminate abnormal information, correct error data, ensure data consistency, and improve data quality. For example, in a human resource management information system, a large amount of employee-related data is stored, and the significance of data quality becomes particularly important. Different types of data sources often appear in different forms. Through the data governance model of the data immunization, the problematic data can be changed into clean data, and the quality of the data itself can be improved. The problem data in the big data is found in advance by using the formulated rules, and then the problem data is automatically recovered.

[0052] Figure 6 The principle flowchart of the data immunization method according to the embodiment of the present application is schematically shown. As shown in Figure 6 The data immunization method in the embodiment automatically recovers the problem data through the data immunization system, including writing programs, scripts, rules and algorithms, etc., and liberates people from the complicated data processing process, but the implementation process is relatively complex. When the data immunization recovery program, script, algorithm, rule cannot process the data, manual intervention and other methods are needed for data processing, which can reduce the complexity of the program rules, reduce a large number of manual processing operations. In this case, we can record the abnormal situation as a data analysis report, similar to the treatment of patients by doctors, medical devices and medicines. In this way, personnel can handle the corresponding processing in the later stage according to the report, which can save a part of the labor work and improve the effect of data immunization.

[0053] The following embodiments are described by taking the immune process of missing data as an example, and mainly relate to the procedures, rules and algorithms of data immunity. Missing data is partial information missing or damaged in data, for example, the information table of human resources staff and the information table of work resume do not match, the staff address information is missing or damaged, etc. For the same staff number, the main table records Zhang San, the work resume records Zhang Wu, and the staff address is incomplete, etc. These data will be filtered out from big data, and according to the difference of missing content, they will be written into different procedures, rules and algorithms, and then form data immunity. When the system encounters these problem data, it can automatically complete and repair according to the refined data immunity framework, immunity rules and algorithms within a specified time, and then write it into the database and automatically form an analysis report for the user to check. For missing value immunity, it can be divided into two categories: 1. Ignore incomplete data. The problem data without business impact can be ignored by directly deleting records and / or attributes. This method has high execution efficiency and can be used as a common method, but if the problem data is large, if part of the record is deleted, the big data system may not be applicable, and the reliability and reliability are reduced. 2. Imputation algorithm, missing value data immunity based on filling technology. In most scenarios, it is more inclined to fill the problem data with the data closest to the missing value to replace the problem data and ensure the quantity and quality of the analyzable data. The imputation method retains potentially useful data, obtains more data samples, is not prone to data bias, and the model construction is more reliable and convincing. Common algorithms generally include statistical methods and classification and clustering methods. For example: in terms of statistics, the personnel information table has an empty age, which is filled by the average age of the institution, which can improve the effectiveness of the data. But this method is more limited; then if the classification and clustering algorithms of data mining are used, the complete data and the similarity between the problem data are used to construct a data model by using machine learning related technologies, such as: according to the classification and clustering algorithm, it is found that the employees of the same batch enter the age close to the average age of the same batch, and then the system can fill in the missing value data according to the average age of the same batch, which is closer to the actual value, and the deviation is small or even close to the actual value, which is beneficial to the subsequent data decision and analysis of big data application.

[0054] Based on the same inventive concept, the application also provides a big model-based data immunity construction device. Figure 7 The structure of the big model-based data immunity construction device according to the embodiment of the application is schematically shown. As Figure 7As shown, a large model-based data immunity construction apparatus includes: a data acquisition module configured to acquire basic data of a target field; an immunity detection module configured to determine an immunity detection rule for detecting problem data based on a problem type of the problem data in the basic data; a problem data module configured to extract data features of the basic data, and obtain a problem data set based on the data features and the immunity detection rule; an immunity repair module configured to construct an immunity algorithm rule for repairing the problem data based on the problem data set, the immunity algorithm rule including large model-based normalization processing; and an immunity output module configured to form data immunity applied to the target field based on the immunity detection rule and the immunity algorithm rule.

[0055] In some optional embodiments of the present application, the basic data includes data recorded by a database or a data table.

[0056] In some optional embodiments of the present application, the problem data includes missing data, duplicate data, error data, and unusable data, and correspondingly, the problem type includes data missing, data duplication, data error, and data unavailability.

[0057] In some optional embodiments of the present application, the data features include values and attributes of the data, and the immunity detection rule includes data constraints, business rule constraints, and data operation constraints; the extracting of the data features of the basic data and the obtaining of the problem data set based on the data features and the immunity detection rule include: adding data in an intermediate state detected by a database transaction to the problem data set; adding duplicate operation result data obtained based on a duplicate operation to the problem data set; adding empty data obtained based on data check missing to the problem data set; adding format error data based on data entry errors to the problem data set; and adding character error data based on data entry errors to the problem data set.

[0058] In some optional embodiments of the present application, the constructing of the immunity algorithm rule for repairing the problem data based on the problem data set includes: determining a data operation in the immunity algorithm rule according to a problem type of the problem data, the data operation including one of supplementing, deleting, format converting, correcting, and matching; and determining an operation parameter in the immunity algorithm rule according to the immunity detection rule corresponding to the problem type, the operation parameter including a value for supplementing, a deletion target, a target format, a corrected value, and a matching result.

[0059] In some optional embodiments of the present application, the data operation in the immune algorithm rule is determined according to the problem type of the problem data, including: if the problem type is data missing, determining that the data operation is supplement; if the problem type is data duplication, determining that the data operation is deletion; if the problem type is data error, determining that the data operation is format conversion when there is format error, and determining that the data operation is correction when there is value error; and if the problem type is data unavailability, determining that the data operation is matching.

[0060] In some optional embodiments of the present application, the operation parameter in the immune algorithm rule is determined according to the immune detection rule corresponding to the problem type, including: generating a supplemented value with a data format constraint within a numerical range constraint based on the data constraint in the immune detection rule; or determining a deletion target from duplicate data based on the business rule constraint in the immune detection rule, and determining a deletion target from duplicate data based on the generation time of the duplicate data if the immune detection rule does not include the business rule constraint; or generating a target format based on the data format constraint in the data constraint of the immune detection rule, and generating a corrected value based on the numerical range constraint in the data constraint of the immune detection rule; if the data error includes data inconsistency, determining a corrected value from inconsistent data based on the trained artificial intelligence algorithm; or generating a matching result based on a combined model obtained by combining a large model and a small model.

[0061] In some optional embodiments of the present application, the trained artificial intelligence algorithm is obtained by the following steps: determining the data dimension of the input of the artificial intelligence algorithm; extracting entities related to the data dimension and entity relationships between entities from the basic data; obtaining extracted data based on the data dimension, entity labels of entities, and entity relationship types using an extraction model; dividing the extracted data into a training set and a validation set, training the artificial intelligence algorithm using the training set, verifying the trained artificial intelligence algorithm using the validation set, and obtaining the trained artificial intelligence algorithm after verification.

[0062] In some optional embodiments of the present application, the combined model obtained by combining a large model and a small model generates a matching result, including: extracting keywords of the unavailable data; inputting the keywords into the small model for normalization processing, and obtaining a first normalization result if the small model normalization is successful; if the small model normalization fails, inputting the keywords into the large model for normalization processing, and obtaining a second normalization result if the large model normalization is successful, the large model and the small model being distinguished by structural complexity and parameter order of magnitude; and performing standardization checking and version conversion on the first normalization result or the second normalization result to obtain the matching result.

[0063] In some optional embodiments of the present application, the device further comprises: before inputting the small model, searching the keyword in a mapping table, if there is a search result, taking the search result as the matching result, and no longer inputting the keyword into the small model; if there is no search result, inputting the keyword into the small model for normalization processing; and after the first normalized result or the second normalized result is subjected to standardization checking and version conversion to obtain the matching result, storing the keyword and the corresponding matching result in the mapping table.

[0064] In some optional embodiments of the present application, the data immunity applied to the target field comprises the immune detection rule and the immune algorithm rule; the immune detection rule is used to identify problem data from the basic data of the target field; and the immune algorithm rule is used to repair data problems in the problem data.

[0065] In some optional embodiments of the present application, the target field comprises a human resource field, and the basic data of the target field comprises at least one of employee basic information, organization information, performance information, training and development information, work resume information, labor contract information, salary information, various configuration code information, and employee-related dimension indicators.

[0066] In some embodiments, a data immunity device is also provided, which comprises: a data scanning module for obtaining to-be-processed data; an immune processing module for inputting the to-be-processed data into a data governance model with data immunity, the data immunity being obtained according to the foregoing data immunity construction method based on a large model; and a data output module for obtaining data after data governance.

[0067] The specific limitations of each functional module in the foregoing data immunity construction device based on a large model and data immunity device can be referred to the limitations of the data immunity construction method based on a large model and / or the data immunity method in the foregoing, which will not be repeated here. Each module in the foregoing device or system can be realized by software, hardware, and combinations thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module. The device realizes the construction and use of data immunity, and also has the advantage of improving data quality.

[0068] The embodiments of the present application provide a storage medium having a program stored thereon, the program being executed by a processor to implement the foregoing data immunity construction method based on a large model or the data immunity method.

[0069] In an embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 1. Figure 8 Figure 8 An internal structure diagram of a computer device according to an embodiment of the present application is schematically shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure), and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown in the figure). The internal memory A03 provides an environment for the operating system B01 and the computer program B02 in the non-volatile storage medium A04 to run. The network interface A02 of the computer device is configured to communicate with an external terminal through a network connection. The computer program B02 is executed by the processor A01 to implement a large model-based data immunity construction method or a data immunity method.

[0070] Those skilled in the art can understand that Figure 8 the structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. Specifically, the computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0071] In an embodiment, the large model-based data immunity construction apparatus provided by the present application can be implemented in the form of a computer program, which can run on a computer device as shown in FIG. 1. The memory of the computer device can store various program modules that constitute the large model-based data immunity construction apparatus. The computer program constituted by the various program modules causes the processor to perform the steps of the large model-based data immunity construction method or the data immunity method of each embodiment of the present application described in the specification. Figure 8

[0072] Those skilled in the art will appreciate that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) containing computer-usable program code.

[0073] ​​The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0074] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0075] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0076] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0077] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as read only memory (ROM) for storing structural information and / or instruction code. Both volatile and non-volatile memory can be implemented as a flash memory, a magnetic memory, an optical memory, and / or any non-transitory computer readable storage medium.

[0078] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0079] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0080] The above is only an embodiment of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for constructing data immunity based on a large model, characterized in that, The method comprises: acquiring basic data of a target field; determining an immune detection rule for detecting problem data based on a problem type of the problem data in the basic data; extracting data features of the basic data, and obtaining a problem data set based on the data features and the immune detection rule; constructing an immune algorithm rule for repairing the problem data based on the problem data set, the immune algorithm rule comprising normalization processing based on a large model; forming data immunity applied to the target field based on the immune detection rule and the immune algorithm rule.

2. The method of claim 1, wherein, The basic data comprises data recorded by a database or a data table.

3. The method of claim 1, wherein, The problem data comprises missing data, duplicate data, error data, and unusable data. Correspondingly, the problem type comprises data missing, data duplication, data error, and data unavailability.

4. The method of claim 1, wherein, The data features comprise values and attributes of the data, and the immune detection rule comprises data constraints, business rule constraints, and data operation constraints. Extracting the data features of the basic data, and obtaining the problem data set based on the data features and the immune detection rule, comprises: adding data in an intermediate state detected by a database transaction to the problem data set; adding duplicate operation result data obtained based on a duplicate operation to the problem data set; adding empty data obtained based on data check missing to the problem data set; adding format error data based on data entry mistakes to the problem data set; adding character error data based on data entry mistakes to the problem data set.

5. The method of claim 3, wherein, Constructing the immune algorithm rule for repairing the problem data based on the problem data set, comprises: determining a data operation in the immune algorithm rule according to the problem type of the problem data, the data operation comprising one of supplement, deletion, format conversion, correction, and matching; determining an operation parameter in the immune algorithm rule according to the immune detection rule corresponding to the problem type, the operation parameter comprising a supplemented value, a deletion target, a target format, a corrected value, and a matching result.

6. The method of claim 5, wherein, Determining the data operation in the immune algorithm rule according to the problem type of the problem data, comprises: if the problem type is data missing, determining that the data operation is supplement; if the problem type is data duplication, determining that the data operation is deletion; if the problem type is data error, determining that the data operation is format conversion when there is format error, and determining that the data operation is correction when there is value error; if the problem type is data unavailability, determining that the data operation is matching.

7. The method of claim 6, wherein, Determining the operation parameter in the immune algorithm rule according to the immune detection rule corresponding to the problem type, comprises: generating a supplemented value with data format constraints within a numerical range constraint based on the data constraints in the immune detection rule; or determining a deletion target from duplicate data based on business rule constraints in the immune detection rule, and determining a deletion target from duplicate data based on generation time of the duplicate data if the immune detection rule does not comprise business rule constraints; or The data format constraint in the data constraint based on the immune detection rule generates a target format, and the numerical range constraint in the data constraint based on the immune detection rule generates a corrected value; if the data error includes non-uniform data, a corrected value is determined from the non-uniform data based on the trained artificial intelligence algorithm; or A combined model obtained by combining a large model and a small model generates a matching result.

8. The method of claim 7, wherein, The trained artificial intelligence algorithm is obtained by the following steps: Determine the data dimension of the input of the artificial intelligence algorithm; Extract entities related to the data dimension and entity relationships between the entities from the basic data; An extracted data is obtained based on the data dimension, entity label of the entity and entity relationship type by using an extraction model; The extracted data is divided into a training set and a verification set, the training set is used to train the artificial intelligence algorithm, the verification set is used to verify the trained artificial intelligence algorithm, and the trained artificial intelligence algorithm is obtained after verification.

9. The method of claim 7, wherein, A combined model obtained by combining a large model and a small model generates a matching result, including: Extracting keywords of unusable data; Input the keywords into the small model for normalization processing, if the small model normalization is successful, a first normalization result is obtained; If the small model normalization fails, input the keywords into the large model for normalization processing, if the large model normalization is successful, a second normalization result is obtained, the large model and the small model are distinguished by structure complexity and parameter order of magnitude; The first normalization result or the second normalization result is subjected to standardization checking and version conversion to obtain the matching result.

10. The method of claim 9, wherein, The method further includes: Before inputting the small model, the keywords are searched in a mapping table, if there is a search result, the search result is taken as the matching result, and the keywords are not input into the small model; If there is no search result, the keywords are input into the small model for normalization processing; and After the first normalization result or the second normalization result is subjected to standardization checking and version conversion to obtain the matching result, the keywords and the corresponding matching result are stored in the mapping table.

11. The method of claim 1, wherein, The data immunity applied to the target field includes the immune detection rule and the immune algorithm rule; The immune detection rule is used to identify problem data from the basic data of the target field; The immune algorithm rule is used to repair data problems in the problem data.

12. The method according to any one of claims 1 to 11, characterized in that, The target field includes the human resource field, and the basic data of the target field includes at least one of employee basic information, organization information, performance information, training and development information, work resume information, labor contract information, salary information, various configuration code information, and employee-related dimension indicators.

13. A data immunization method, characterized by, The method includes: Obtaining data to be processed; Input the data to be processed into a data governance model with data immunity, the data immunity is obtained according to the data immunity construction method based on the large model of any one of claims 1 to 12; Obtain the data after data governance. 14.A large model-based data immunity construction apparatus, characterized in that, The device includes: A data acquisition module for acquiring basic data of a target field; An immune detection module is configured to determine immune detection rules for detecting problem data based on a problem type of the problem data in the basic data; A problem data module is configured to extract data features of the basic data, and obtain a problem data set based on the data features and the immune detection rules; An immune repair module is configured to construct immune algorithm rules for repairing the problem data based on the problem data set, the immune algorithm rules including normalization processing based on a large model; and An immune output module is configured to form data immunity applied to the target field based on the immune detection rules and the immune algorithm rules.

15. A data immune device, comprising: The device comprises: a data scanning module configured to obtain to-be-processed data; an immune processing module configured to input the to-be-processed data into a data governance model having data immunity, the data immunity being obtained according to the large model-based data immunity construction method of any one of claims 1 to 12; and a data output module configured to obtain data processed by data governance.

16. A computer-readable storage medium, the storage medium storing instructions, when executed on a computer, causing the computer to perform the large model-based data immunity construction method of any one of claims 1 to 12 or the data immunity method of claim 13.

17. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the large model-based data immunity construction method of any one of claims 1 to 12 or the data immunity method of claim 13. The computer program, when executed by a processor, implements the large model-based data immunity construction method of any one of claims 1 to 12 or the data immunity method of claim 13.

Images