A data processing method, electronic device and computer storage medium

Abstract

The embodiments of the present application disclose a data processing method, an electronic device and a computer storage medium, which relate to data processing and data standardization technology, wherein the method comprises: obtaining original data and data labels carried by the original data; performing desensitization processing on the original data to obtain target data; obtaining standardization rules corresponding to the data labels, wherein the standardization rules comprise integrity check rules and format rules; processing the target data according to the standardization rules to obtain standard data satisfying the integrity check rules and the format rules, so as to facilitate the standardization of data for subsequent data storage or processing, thereby improving the efficiency and accuracy of data processing.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, an electronic device and a computer storage medium. Background Art

[0002] Big data has been widely regarded as a strategic new resource that can define the massive data generated in today's era and the related technological development and service innovation. Big data contains huge commercial value. In the field of data circulation, with the rapid growth of data volume, the data quality of different data sources is uneven. There are different concepts and standards for data quality. Data quality generally includes quantitative descriptions such as data accuracy, completeness, timeliness, and consistency. For users in the era of big data, the amount of data they need to store and process is large, and the data sources and data structures are numerous and complex, which brings many challenges to the analysis and application of big data. If users want to give full play to the opportunities and advantages given by big data, they must have reliable, accurate, and timely high-quality data. Only by extracting implicit and useful information from high-quality large-scale data can they make more accurate decisions that are more in line with market and customer needs. For this reason, users pay more attention to data quality and its importance.

[0003] In the field of healthcare, a large amount of medical insurance data needs to be processed in all aspects, so data processing still needs to improve processing efficiency while ensuring accuracy. The data itself comes from a complex source, there may be certain errors, and its data type and format may vary greatly. Therefore, in the process of data processing and analysis, the analysis results cannot be accurately obtained, and errors or even mistakes may occur in the system, and the stability and efficiency of data processing are low. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, an electronic device, and a computer storage medium, which involve data processing and data standardization technologies, and facilitate obtaining standardized data for subsequent data storage or processing processes, thereby improving the efficiency and accuracy of data processing.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0006] Acquire original data and data labels carried by the original data;

[0007] Performing desensitization processing on the original data to obtain target data;

[0008] Obtaining standardization rules corresponding to the data tags, wherein the standardization rules include integrity check rules and format rules;

[0009] The target data is processed according to the standardization rule to obtain standard data that meets the integrity check rule and the format rule.

[0010] As a possible implementation manner, in response to the standardization rule including the integrity check rule, processing the target data according to the standardization rule includes:

[0011] Detect whether there are any vacancies in the target data, and if so, generate a report containing the vacancies, and output the report to prompt the completion of the vacancies.

[0012] As a possible implementation manner, the standardization rule further includes a duplicate elimination rule, and the processing of the target data according to the standardization rule further includes:

[0013] The similarity of the items in the target data is detected, and if the similarity is higher than a first threshold, the items are determined to be duplicate items, and the data of the duplicate items are merged or the data of one of the duplicate items is deleted.

[0014] As a possible implementation manner, in response to the standardization rule including the format rule, processing the target data according to the standardization rule further includes:

[0015] Check whether the project field in the target data matches the template field. If not, convert the unmatched project field into a target field that matches the template field according to the format rule.

[0016] As a possible implementation, the method further includes:

[0017] If it is detected that the project field that does not match the template field contains erroneous data, an error record is generated;

[0018] Determine the item category of the item field, search for correction data corresponding to the item category and matching the item field in a correction database, and output prompt information containing the correction data.

[0019] As a possible implementation manner, after obtaining the standard data satisfying the integrity check rule and the format rule, the method further includes:

[0020] Obtaining a pre-stored data template corresponding to the data tag, performing data comparison between the standard data and the pre-stored data template, and obtaining a data score;

[0021] It is determined whether the data score is higher than a first score threshold, and if so, the standard data is grouped and stored.

[0022] As a possible implementation manner, after obtaining the standard data satisfying the integrity check rule and the format rule, the method further includes:

[0023] Generate a data detection report, the data detection report including the data detection time of the original data, the data repetition rate, the data qualification rate and / or the content of processing the target data according to the standardization rule, wherein the data repetition rate is the proportion of repeated data in the target data, and the data qualification rate is the proportion of standard data in the target data.

[0024] In a second aspect, an embodiment of the present application provides an electronic device, including: an acquisition module, a desensitization module and a data processing module, wherein:

[0025] The acquisition module is used to acquire the original data and the data tags carried by the original data;

[0026] The desensitization module is used to perform desensitization processing on the original data to obtain target data;

[0027] The acquisition module is also used to acquire the standardization rules corresponding to the data tags, wherein the standardization rules include integrity check rules and format rules;

[0028] The data processing module is used to process the target data according to the standardization rules to obtain standard data that meets the integrity check rules and the format rules.

[0029] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, an input device, an output device and a memory, wherein the processor, input device, output device and memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method described in the first aspect and any possible implementation manner thereof.

[0030] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method of the above-mentioned first aspect and any possible implementation method thereof.

[0031] The embodiment of the present application obtains the original data and the data labels carried by the original data, desensitizes the original data, obtains the target data, and then obtains the standardization rules corresponding to the data labels, wherein the standardization rules include integrity check rules and format rules. The target data is then processed according to the standardization rules to obtain standard data that meets the integrity check rules and the format rules, so as to facilitate the standardization of the data for subsequent data storage or processing, thereby improving the efficiency and accuracy of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0033] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0034] Figure 2 is a flowchart of a data processing method provided by another embodiment of the present application;

[0035] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0036] Figure 4 It is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0039] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0040] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0041] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0042] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0043] In order to better understand the embodiments of the present application, the method of applying the embodiments of the present application will be introduced below.

[0044] The electronic devices mentioned in the embodiments of the present application may include various handheld devices with wireless communication functions, vehicle-mounted devices, wearable devices (such as smart watches, smart bracelets, pedometers, etc.), computing devices or other processing devices connected to wireless modems, and various forms of user equipment (UE), mobile stations (MS), terminal devices, etc. For the convenience of description, the above-mentioned devices are collectively referred to as electronic devices.

[0045] See also Figure 1 , is a schematic flow chart of a data processing method provided in an embodiment of the present application, and the method can be applied to the above-mentioned electronic devices, such as Figure 1 The method may include:

[0046] 101. Obtain original data and data labels carried by the original data.

[0047] The original data in the embodiments of the present application may be medical insurance data. Medical insurance generally refers to basic medical insurance, which is a social insurance system established to compensate workers for economic losses caused by disease risks. A medical insurance fund is established through employers and individuals paying contributions. When insured persons incur medical expenses due to illness and treatment, the medical insurance agency will provide them with certain financial compensation. The above process involves a large amount of medical insurance data processing.

[0048] The above-mentioned original data may be data in various types of documents, such as data in files generated by word editing software such as WORD and EXCEL.

[0049] The electronic device can communicate with the terminal device and receive original data from the terminal device. After receiving the above original data, step 101 can be executed immediately, and the electronic device can store a preset execution time, that is, the user can set the data processing time in advance, and then the above step 101 can be executed at the above preset execution time.

[0050] In order to process the data quickly and in a standardized manner, the embodiment of the present application uses a method for classifying the above-mentioned raw data. The above-mentioned raw data may carry different data labels due to different sources, and the above-mentioned raw data may be classified and processed through the above-mentioned data labels. The above-mentioned data labels may be divided according to the data type, for example, different hospitals report data in different formats to perform data standardization and establish a database; each format corresponds to a data label, and its data format can be determined according to the above-mentioned data label.

[0051] 102. Perform desensitization processing on the above raw data to obtain target data.

[0052] Data desensitization in the embodiments of the present application refers to the deformation of certain sensitive information through desensitization rules to achieve reliable protection of sensitive privacy data. In the case of customer security data or some commercial sensitive data, real data is transformed and provided for testing without violating system rules. Personal information such as ID card number, mobile phone number, card number, customer number, etc. need to be desensitized. One of the database security technologies, database security technology mainly includes: database leakage scanning, database encryption, database firewall, data desensitization, and database security audit system. Database security risks include: dragging database, brushing database, and hitting database.

[0053] In the embodiment of the present application, desensitization rules can be pre-set, that is, desensitization of certain specific types of data can be set. The above desensitization rules can be set according to actual conditions, and the types of data that need to be desensitized can be selected. The main desensitization rules in the embodiment of the present application may include: name, ID number, address information, and telephone number.

[0054] In the medical insurance field, data desensitization can improve data security and protect user privacy.

[0055] After desensitization, step 103 can be executed, and the target data can be specifically standardized.

[0056] 103. Obtain the standardization rules corresponding to the above data labels.

[0057] Before data analysis, we usually need to normalize the data first, and use the normalized data for data analysis. Data normalization is the indexation of statistical data. Data normalization mainly includes two aspects: data homogeneity and dimensionless processing. Data homogeneity mainly solves the problem of data of different properties. Directly summing up indicators of different properties cannot correctly reflect the comprehensive results of different forces. It is necessary to first consider changing the properties of the inverse indicator data so that the forces of all indicators on the evaluation scheme are homogeneous, and then summing them up to get the correct results. Data dimensionless processing mainly solves the comparability of data. There are many methods for data normalization, and the commonly used ones are "minimum-maximum normalization", "Z-score normalization" and "normalization by decimal calibration". After the above normalization, the original data are converted into dimensionless indicator evaluation values, that is, the values ​​of each indicator are at the same level, and comprehensive evaluation and analysis can be carried out.

[0058] Mainly due to the continuous increase in data volume, the original data structure may be unreasonable and cannot meet various requirements. The replacement of the database and the data structure requires the conversion of the data itself.

[0059] Data transcoding, also known as data conversion, is the process of changing data from one form of representation to another. In the data transfer of the medical insurance system, there is a change in the database and the data structure, which requires the conversion of the data itself.

[0060] The correspondence between the above data labels and standardization rules can be understood as a data conversion standard, and standardized data transcoding is achieved through the above data conversion standard.

[0061] Data conversion standards include a set of plans for encoding data according to field, record and file requirements so that it can be converted through specified media. The data model is a prerequisite for developing encoding rules, and the intermediary nature of conversion standards is a major feature. After the conversion standards are optimized, all data can be effectively communicated, and after the products and database structures are optimized, they can be effectively stored, applied and maintained.

[0062] The electronic device may store a correspondence between data tags and standardization rules, that is, data with different data tags are processed using different standardization rules. The standardization rules to be used may be determined through the data tags, and then the standardization rules may be obtained and step 104 may be executed.

[0063] 104. Process the target data according to the standardization rules to obtain standard data that meets the integrity check rules and the format rules.

[0064] Since the original data comes from different sources, the data transcoding process can be performed in a targeted manner.

[0065] Specifically, the electronic device pre-stores standardization rules for standardizing data, which may be multiple rules, and stores the correspondence between data tags and standardization rules. The standardization rules corresponding to the data tags carried by the original data can be obtained through the correspondence between the data tags and the standardization rules, and then the transcoding process is performed using the standardization rules.

[0066] Data cleaning is the process of reviewing and verifying data in order to remove duplicate information, correct existing errors, and provide data consistency.

[0067] The above-mentioned standardization rules may include integrity check rules, which are used to detect the integrity of the target data, detect missing or omitted data, and complete the missing or omitted data according to the standardization rules.

[0068] The above-mentioned standardization rules may include duplicate detection rules, which are used to detect duplicate data in the target data and delete the duplicate data.

[0069] The above-mentioned standardization rules may include format rules, and the format of the target data may be unified according to the above-mentioned format rules.

[0070] The above standardization rules can also include the processing of erroneous data. The cause of the error may be that the business system is not sound enough, and it is directly written into the backend database without judgment after receiving the input, such as the input of numerical data as full-width numeric characters, a carriage return operation after the string data, incorrect date format, date out of range, etc. This type of data must also be classified. For problems such as full-width characters and invisible characters before and after the data, they can only be found by writing SQL statements, and then the customer is required to extract after the business system is corrected. Errors such as incorrect date format or date out of range will cause ETL operation failure. This type of error can be picked out in the business system database using SQL, and handed over to the business department for correction within a time limit, and then extracted after correction.

[0071] The target data is processed according to the above-mentioned standardization rules, mainly by standardization processing through integrity checking rules and format rules, so that standard data that meets the above-mentioned integrity checking rules and the above-mentioned format rules can be obtained.

[0072] Optionally, after obtaining the standard data, a subsequent data processing flow may be performed on the standard data.

[0073] Optionally, the above standard data may be stored in the first space.

[0074] The above-mentioned standardized standard data can be recognized by the system after standardization, and data analysis and processing can be performed using these standard data, and they can be stored in a preset space for preservation, so as to facilitate subsequent query or call.

[0075] The embodiment of the present application obtains the original data and the data labels carried by the original data, desensitizes the original data, obtains the target data, and then obtains the standardization rules corresponding to the data labels, wherein the standardization rules include integrity check rules and format rules. The target data is then processed according to the standardization rules to obtain standard data that meets the integrity check rules and the format rules, so as to facilitate the standardization of the data for subsequent data storage or processing, thereby improving the efficiency and accuracy of data processing.

[0076] See also Figure 2 , is a schematic flow chart of another data processing method provided in an embodiment of the present application, Figure 2 The embodiment shown may be Figure 1 Based on the embodiment shown, Figure 2 The method may include:

[0077] 201. Obtain original data and data labels carried by the original data.

[0078] The above step 201 can refer to Figure 1The specific description of step 101 in the illustrated embodiment will not be repeated here.

[0079] 202. Perform desensitization processing on the above raw data to obtain target data.

[0080] The above step 202 can refer to Figure 1 The specific description of step 102 in the illustrated embodiment will not be repeated here.

[0081] 203. Obtain the standardization rules corresponding to the above data labels.

[0082] The above-mentioned standardized rules may include integrity check rules, duplicate check rules and / or format rules, etc. If the integrity check rules are included, step 204 can be executed; if the duplicate check rules are included, step 205 can be executed; if the format rules are included, step 206 can be executed.

[0083] The above step 203 can refer to Figure 1 The specific description of step 103 in the illustrated embodiment will not be repeated here.

[0084] 204. In response to the above-mentioned standardization rules including completeness check rules, whether there are any vacancies in the items in the above-mentioned target data, if there are any vacancies, a report including the above-mentioned vacancies is generated, and the above-mentioned report is output to prompt the above-mentioned vacancies to be completed.

[0085] When the standardized rules corresponding to the above-mentioned data labels obtained include integrity checking rules, the integrity of the target data can be checked, and whether there are vacancies in the items in the above-mentioned target data can be detected. Vacancies can be understood as unfilled items in the data. This type of data mainly includes some missing information that should be there, such as the name of the supplier, the name of the branch, the regional information of the customer, and the main table and the detail table in the business system cannot match. This type of missing data can be filtered out and written into the data warehouse after being completed according to the integrity checking rules. Specifically, if there are vacancies, the above-mentioned vacant items can be sorted out, a report containing the above-mentioned vacant items can be generated, and the report can be output to prompt the user to complete the above-mentioned vacant items. For example, if there is no data (not filled in) for the item recording the admission time in the report, the above-mentioned report can be sent to the source of the above-mentioned original data to request the filling of the above-mentioned vacant items. Optionally, some missing values ​​can be derived from this data source or other data sources, so that the missing values ​​can be replaced by the average value, maximum value, minimum value or more complex probability estimation, thereby achieving the purpose of cleaning. Therefore, the above-mentioned integrity check rules may also include completion rules, and the above-mentioned blank items may also be completed through the above-mentioned completion rules. For example, the admission time contained in the target data is not filled in, but other relevant information of the user, such as electronic medical records, can be obtained in the electronic device, and the electronic device can obtain the admission time recorded in the electronic medical record. Then, the electronic device can complete the admission time in the above-mentioned target data. The process can also generate a completion record, and the completed information can be queried.

[0086] 205. Detect the similarity of the items in the target data. If the similarity is higher than a first threshold, determine that the item is a duplicate item, and delete data of one of the duplicate items.

[0087] When the above-mentioned standardization rules include duplicate detection rules, duplicate data in the target data can be detected to achieve data deduplication.

[0088] Optionally, records in the database with the same attribute values ​​are considered duplicate records. Whether the records are equal is detected by judging whether the attribute values ​​between the records are equal, and equal records are merged into one record (i.e., merged / cleared). The electronic device can detect the similarity of the items in the above target data, and obtain the above similarity by comparing the fields in the data. The electronic device can also store a first threshold value. When the similarity is higher than the first threshold value, the above item can be judged as a duplicate item, and the data of the above duplicate item can be deleted. For example, if the first threshold value is set to 98%, and there are two items with a similarity of 98.7% in the target data detected by the electronic device, the data of one of the items is deleted, or the data of the above duplicate items is merged into one copy, and the above analysis record is retained and stored in the electronic device.

[0089] Optionally, a second threshold may also be stored in the electronic device. When the similarity is lower than the first threshold and higher than the second threshold, the item may be judged as a high-similarity item and then stored in a high-similarity record for user query and reference. It may also be manually determined whether it is duplicate data, thereby improving the accuracy of data processing.

[0090] In the embodiment of the present application, for massive medical insurance data, storage space can be saved by deduplicating data. By deduplicating data, the number of computer storage media required can be greatly reduced, thereby reducing costs. It can even make the cost of hard disk-based storage systems lower than that of tape libraries while providing better performance. Therefore, a storage system that supports data deduplication technology is particularly suitable for data backup.

[0091] Data deduplication can also improve write performance. The write performance of a disk is limited, usually about 100MB / s for sequential writes. If data deduplication is performed when writing data, some data can be avoided from being written to the disk, thereby improving write performance.

[0092] Data deduplication can also save network bandwidth. If data deduplication is performed on the client, only the newly added data is transferred to the storage system, which can reduce the amount of data transmission on the network and save network bandwidth.

[0093] 206. In response to the above-mentioned standardization rules including format rules, whether the project fields in the above-mentioned target data match the template fields is detected. If they do not match, the above-mentioned unmatched project fields are converted into target fields that match the above-mentioned template fields according to the above-mentioned format rules.

[0094] Specifically, statistical analysis methods can be used to identify possible erroneous values ​​or outliers, such as deviation analysis, identifying values ​​that do not conform to distribution or regression equations, or to check data values ​​using a simple rule library (common sense rules, business-specific rules, etc.), or to use constraints between different attributes or external data to detect and clean data.

[0095] Multiple template fields can be stored in an electronic device. For different types of data, the project fields in the above target data can be compared with the template fields. If the above project fields conform to the format of the above template fields, they are matched, otherwise they are not matched. For example, for the date of consultation data, the date of consultation is March 11, 2017, and the data format of the consultation date item in the target data is "2017.3.11", while the format rule for the consultation date is "XXXX (year) -XX (month) -XX (day)" (template field). The electronic device can automatically identify the above consultation date data "2017.3.11", compare it with the template field, detect that its format does not match the template field, and convert it to the matching format "2017-03-11" to achieve the effect of unified data format.

[0096] If there is no match, the unmatched project fields may be converted into target fields that match the template fields according to the format rules.

[0097] 207. If it is detected that the above-mentioned project fields that do not match the template fields contain erroneous data, an error record is generated.

[0098] In some cases, the reason why the project field and the template field do not match is not a format problem, but an input data error. For example, the same is the consultation date data. The consultation date is March 11, 2017, and the consultation date project data in the target data is "2017.3.111". The electronic device can detect that the value range of the target data is wrong. "111" can be judged as the date of a certain month, but it exceeds the normal date range (31 days in January). It can be judged that there is erroneous data in the above-mentioned project field, and such erroneous data cannot determine the correct data. Therefore, the above-mentioned error record can be generated, and it can be recorded in the report for the user to view, and the above-mentioned error record can be output for prompting, so as to facilitate the correction of data and obtain accurate data.

[0099] The above steps 204 to 206 may be executed in any order or simultaneously.

[0100] 208. Determine the project category of the project field, search for correction data matching the project field in a correction database corresponding to the project category, and output prompt information containing the correction data.

[0101] For different erroneous data, they can be automatically corrected in some cases. The electronic device can store a correction database, which contains correction data corresponding to different project categories. The electronic device can determine the project category of the above project field, and then search the correction data corresponding to the project category and matching the above project field in the above correction database. Furthermore, it can output prompt information containing the above correction data to suggest the user to modify the above erroneous data.

[0102] For example, if the transfer department item field in the target data is filled in incorrectly (such as a typo), it can be determined that the item category corresponding to the transfer department is the hospitalization process information, and the correction data corresponding to the hospitalization process information is obtained in the correction database, which includes all departments. By identifying the above erroneous data, the correction data can be determined, that is, the correct department name, such as dermatology, neurosurgery, etc.

[0103] After processing the target data through the above steps, standard data satisfying the above integrity check rules and the above format rules can be obtained.

[0104] Optionally, the method may further include: obtaining a pre-stored data template corresponding to the data tag, performing data comparison between the standard data and the pre-stored data template to obtain a data score;

[0105] It is determined whether the data score is higher than a first score threshold. If so, the standard data is grouped and stored.

[0106] Specifically, it can be determined whether the standard data meets the data quality requirements. Pre-stored data templates for different data can be stored in the electronic device. The pre-stored data template corresponding to the data can be determined through the above data tag. The pre-stored data template may include data structure templates, data format templates and other contents, and also include corresponding data scoring rules, such as 1 point deduction for data format errors, 0.5 points deduction for a data item not filled in, etc. After obtaining the above pre-stored data template, the above standard data can be compared with the above pre-stored data template, and the standard data can be scored to obtain the above data score. Among them, data comparison and scoring can be performed on all target data uploaded at one time, and data comparison and scoring can also be performed in groups.

[0107] After obtaining the data score, it can be determined whether the data score is higher than a first score threshold. If the data score is higher than the first score threshold, the data quality requirement is met and the data can be stored in the standard data group.

[0108] Optionally, the electronic device may further store multiple score thresholds, and the quality level of the standard data may be determined by comparing with the above data score. For example, a second score threshold higher than the above first score threshold may be set, and it may be determined whether the above data score is higher than the second score threshold. If the above data score is higher than the second score threshold, the above data quality requirements are met and the data quality level is excellent. If the above data score is higher than the above first score threshold but not higher than the above second score threshold, the above data quality requirements are met but the data quality level is good.

[0109] The grouping basis can be based on data quality level or data type, and there is no limitation here.

[0110] By evaluating the data quality level, the data can be sorted according to the data quality situation, and the data quality situation can be intuitively understood, making it easier to refer to the data quality to perform subsequent data processing steps.

[0111] Optionally, the method further includes: generating a data detection report, wherein the data detection report includes the data detection time, data repetition rate, data qualification rate of the original data and / or the content of processing the target data according to the standardization rules.

[0112] The data repetition rate is the ratio of repeated data in the target data, and the data qualification rate is the ratio of standard data in the target data. The data detection report can clearly reflect the data quality, provide an effective reference for data acquisition and processing, and facilitate users to maintain data and improve data acquisition and data processing systems.

[0113] The embodiment of the present application obtains the original data and the data tags carried by the original data, performs desensitization processing on the original data, obtains the target data, obtains the standardization rules corresponding to the data tags, and processes the target data according to the standardization rules, including: detecting whether there are vacancies in the items in the target data, if there are vacancies, generating a report containing the vacant items, and outputting the report to prompt the completion of the vacant items; detecting the similarity of the items in the target data, if the similarity is higher than a first threshold, judging that the items are duplicate items, and deleting the data of one of the duplicate items; detecting whether the item field in the target data matches the template field, if not, converting the unmatched item field into a target field matching the template field according to the format rules; if it is detected that the item field that does not match the template field has erroneous data, generating an error record, determining the item category of the item field, searching for the corrected data matching the item field in the correction database corresponding to the item category, and outputting prompt information containing the corrected data, so as to obtain standard data that meets the integrity check rules and the format rules for subsequent data storage or processing, and improve the efficiency and accuracy of data processing.

[0114] See also Figure 3 , Figure 3 3 is a schematic diagram of the structure of an electronic device 300 provided in an embodiment of the present application. The electronic device 300 includes an acquisition module 310, a desensitization module 320 and a data processing module 330, wherein:

[0115] The acquisition module 310 is used to acquire the original data and the data tags carried by the original data;

[0116] The original data in the embodiments of the present application may be medical insurance data. Medical insurance generally refers to basic medical insurance, which is a social insurance system established to compensate workers for economic losses caused by disease risks. A medical insurance fund is established through employers and individuals paying contributions. When insured persons incur medical expenses due to illness and treatment, the medical insurance agency will provide them with certain financial compensation. The above process involves a large amount of medical insurance data processing.

[0117] The above-mentioned original data may be data in various types of documents, such as data in files generated by word editing software such as WORD and EXCEL.

[0118] The electronic device can communicate with the terminal device and receive raw data from the terminal device. After receiving the above raw data, the acquisition module 310 can immediately execute the above steps, and the electronic device can store a preset execution time, that is, the user can set the data processing time in advance, and then the acquisition module 310 can execute the above steps at the above preset execution time.

[0119] In order to process the data quickly and in a standardized manner, the embodiment of the present application uses a method for classifying the above-mentioned raw data. The above-mentioned raw data may carry different data labels due to different sources, and the above-mentioned raw data may be classified and processed through the above-mentioned data labels. The above-mentioned data labels may be divided according to the data type, for example, different hospitals report data in different formats to perform data standardization and establish a database; each format corresponds to a data label, and its data format can be determined according to the above-mentioned data label.

[0120] The desensitization module 320 is used to perform desensitization processing on the original data to obtain target data;

[0121] Data desensitization in the embodiments of the present application refers to the deformation of certain sensitive information through desensitization rules to achieve reliable protection of sensitive privacy data. In the case of customer security data or some commercial sensitive data, real data is transformed and provided for testing without violating system rules. Personal information such as ID card number, mobile phone number, card number, customer number, etc. need to be desensitized. One of the database security technologies, database security technology mainly includes: database leakage scanning, database encryption, database firewall, data desensitization, and database security audit system. Database security risks include: dragging database, brushing database, and hitting database.

[0122] In the embodiment of the present application, desensitization rules can be pre-set, that is, desensitization of certain specific types of data can be set. The desensitization module 320 can set the above desensitization rules according to actual conditions and select the type of data that needs to be desensitized. The main desensitization rules in the embodiment of the present application may include: name, ID number, address information, and telephone number.

[0123] In the medical insurance field, data desensitization can improve data security and protect user privacy.

[0124] After the desensitizing module 320 performs data desensitization, specific standardization processing can be performed.

[0125] The acquisition module 310 is also used to acquire the standardization rules corresponding to the data tags;

[0126] The electronic device may store the correspondence between data tags and standardization rules, that is, data with different data tags are processed using different standardization rules. The acquisition module 310 may determine the standardization rule to be used through the data tag, and then acquire the standardization rule to execute subsequent steps.

[0127] The data processing module 330 is used to process the target data according to the standardization rules to obtain standard data that meets the integrity check rules and the format rules.

[0128] Since the original data comes from different sources, the data transcoding process can be performed in a targeted manner.

[0129] Specifically, the electronic device pre-stores standardization rules for standardizing data, which may be multiple rules, and stores the correspondence between data tags and standardization rules. The data processing module 330 can obtain the standardization rules corresponding to the data tags carried by the original data through the correspondence between the data tags and the standardization rules, and then use the above standardization rules to perform the transcoding process.

[0130] Data cleaning is the process of reviewing and verifying data in order to remove duplicate information, correct existing errors, and provide data consistency.

[0131] The above-mentioned standardization rules may include integrity check rules, which are used to detect the integrity of the target data, detect missing or omitted data, and complete the missing or omitted data according to the standardization rules.

[0132] The above-mentioned standardization rules may include duplicate detection rules, which are used to detect duplicate data in the target data and delete the duplicate data.

[0133] The above-mentioned standardization rules may include format rules, and the format of the target data may be unified according to the above-mentioned format rules.

[0134] The above standardization rules can also include the processing of erroneous data. The cause of the error may be that the business system is not sound enough, and it is directly written into the backend database without judgment after receiving the input, such as the input of numerical data as full-width numeric characters, a carriage return operation after the string data, incorrect date format, date out of range, etc. This type of data must also be classified. For problems such as full-width characters and invisible characters before and after the data, they can only be found by writing SQL statements, and then the customer is required to extract after the business system is corrected. Errors such as incorrect date format or date out of range will cause ETL operation failure. This type of error can be picked out in the business system database using SQL, and handed over to the business department for correction within a time limit, and then extracted after correction.

[0135] The data processing module 330 processes the target data according to the above-mentioned standardization rules, mainly by performing standardization processing through integrity checking and format rules, and can obtain standard data that meets the above-mentioned integrity checking rules and the above-mentioned format rules.

[0136] Optionally, the data processing module 330 includes a first processing unit 331, configured to:

[0137] Check whether there are any vacancies in the target data. If there are any vacancies, generate a report containing the vacancies and output the report to prompt completion of the vacancies.

[0138] When the standardization rules corresponding to the above-mentioned data tags obtained include integrity checking rules, the first processing unit 331 can check the integrity of the target data, and can detect whether there are vacancies in the items in the above-mentioned target data. Vacancies can be understood as unfilled items in the data. This type of data mainly includes some missing information that should be there, such as the name of the supplier, the name of the branch, the regional information of the customer, and the main table and the detail table in the business system cannot match. This type of missing data can be filtered out and written into the data warehouse after being completed according to the integrity checking rules. Specifically, if there are vacancies, the first processing unit 331 can sort out the above-mentioned vacant items, generate a report containing the above-mentioned vacant items, and output a report to prompt the user to complete the above-mentioned vacant items. For example, if there is no data (not filled in) for the item recording the admission time in the report, the above-mentioned report can be sent to the source of the above-mentioned original data to request the filling of the above-mentioned vacant items. Optionally, some missing values ​​can be derived from this data source or other data sources, so that the missing values ​​can be replaced by the average value, maximum value, minimum value or more complex probability estimation, thereby achieving the purpose of cleaning. Therefore, the above-mentioned integrity check rules may also include completion rules, and the above-mentioned blank items may also be completed through the above-mentioned completion rules. For example, the admission time contained in the target data is not filled in, but other relevant information of the user, such as electronic medical records, can be obtained in the electronic device, and the electronic device can obtain the admission time recorded in the electronic medical record. Then, the electronic device can complete the admission time in the above-mentioned target data. The process can also generate a completion record, and the completed information can be queried.

[0139] Optionally, the data processing module 330 includes a second processing unit 332, configured to:

[0140] The similarity of the items in the target data is detected. If the similarity is higher than a first threshold, the items are determined to be duplicate items, and the data of the duplicate items are merged or the data of one of the duplicate items is deleted.

[0141] When the above-mentioned standardization rules include duplicate detection rules, the second processing unit 332 can detect duplicate data in the target data to achieve data deduplication.

[0142] Optionally, records with the same attribute values ​​in the database are considered duplicate records. Whether the records are equal is detected by judging whether the attribute values ​​between the records are equal, and equal records are merged into one record (i.e., merged / cleared). The second processing unit 332 can detect the similarity of the items in the above-mentioned target data, and obtain the above-mentioned similarity by comparing the fields in the data. The electronic device can also store a first threshold value. When the similarity is higher than the first threshold value, the second processing unit 332 can judge that the above-mentioned items are duplicate items, and then delete the data of the above-mentioned duplicate items. For example, if the first threshold value is set to 98%, and there are two items with a similarity of 98.7% in the target data detected by the second processing unit 332, the data of one of the items is deleted, or the data of the above-mentioned duplicate items are merged into one copy, and the above-mentioned analysis record is retained and stored in the electronic device.

[0143] Optionally, the data processing module 330 includes a third processing unit 333, configured to:

[0144] Check whether the project field in the above target data matches the template field. If not, convert the above unmatched project field into a target field that matches the above template field according to the above format rules.

[0145] Specifically, the third processing unit 333 can use statistical analysis methods to identify possible erroneous values ​​or abnormal values, such as deviation analysis, identifying values ​​that do not comply with distribution or regression equations, or use a simple rule library (common sense rules, business-specific rules, etc.) to check data values, or use constraints between different attributes and external data to detect and clean data.

[0146] Multiple template fields can be stored in the electronic device. For different types of data, the third processing unit 333 can compare the project field in the above target data with the template field. If the above project field conforms to the format of the above template field, it is matched, otherwise it is not matched. For example, for the date of consultation data, the date of consultation is March 11, 2017, and the data format of the date of consultation item in the target data is "2017.3.11", and the format rule for the date of consultation is "XXXX (year) -XX (month) -XX (day)" (template field). The third processing unit 333 can automatically identify the above date of consultation data "2017.3.11", compare it with the template field, detect that its format does not match the template field, and convert it into a matching format "2017-03-11" to achieve the effect of unified data format.

[0147] In the embodiment of the present application, for massive medical insurance data, storage space can be saved by deduplicating data. By deduplicating data, the number of computer storage media required can be greatly reduced, thereby reducing costs. It can even make the cost of hard disk-based storage systems lower than that of tape libraries while providing better performance. Therefore, a storage system that supports data deduplication technology is particularly suitable for data backup.

[0148] Data deduplication can also improve write performance. The write performance of a disk is limited, usually about 100MB / s for sequential writes. If data deduplication is performed when writing data, some data can be avoided from being written to the disk, thereby improving write performance.

[0149] Data deduplication can also save network bandwidth. If data deduplication is performed on the client, only the newly added data is transferred to the storage system, which can reduce the amount of data transmission on the network and save network bandwidth.

[0150] Optionally, the data processing module 330 further includes a correction unit 334, which is used to:

[0151] If it is detected that the above-mentioned project fields that do not match the template fields contain erroneous data, an error record is generated;

[0152] Determine the project category of the project field, search for correction data corresponding to the project category and matching the project field in a correction database, and output prompt information containing the correction data.

[0153] In some cases, the reason why the project field and the template field do not match is not a format problem, but an input data error. For example, the same is the consultation date data. The consultation date is March 11, 2017, and the consultation date project data in the target data is "2017.3.111". The correction unit 334 can detect that the value range of the target data is wrong. "111" can be judged as the date of a certain month, but it exceeds the normal date range (31 days in a month). The correction unit 334 can determine that there is erroneous data in the above-mentioned project field, and such erroneous data cannot determine the correct data. Therefore, the above-mentioned error record can be generated, and it can be recorded in the report for the user to view, and the above-mentioned error record can be output for prompting, so as to facilitate the correction of data and obtain accurate data.

[0154] For different erroneous data, they can be automatically corrected in some cases. The electronic device can store a correction database, which contains correction data corresponding to different project categories. The correction unit 334 can determine the project category of the above project field, and then search the correction data corresponding to the project category and matching the above project field in the above correction database. Further, a prompt information containing the above correction data can be output to suggest the user to modify the above erroneous data.

[0155] For example, if the transfer department item field in the target data is filled in incorrectly (such as a typo), the correction unit 334 can determine that the item category corresponding to the transfer department is the hospitalization process information, and obtain the correction data corresponding to the hospitalization process information in the correction database, which includes all departments. The correction unit 334 can determine the correction data, that is, the correct department name, such as dermatology, neurosurgery, etc., by identifying the above erroneous data.

[0156] After processing the target data through the above steps, standard data satisfying the above integrity check rules and the above format rules can be obtained.

[0157] Optionally, the electronic device 300 further includes a scoring module 340, which is used to:

[0158] After obtaining the standard data that meets the integrity check rules and the format rules, obtain the pre-stored data template corresponding to the data tag, compare the standard data with the pre-stored data template, and obtain a data score;

[0159] It is determined whether the data score is higher than a first score threshold. If so, the standard data is grouped and stored.

[0160] Specifically, the scoring module 340 can determine whether the standard data meets the data quality requirements. Pre-stored data templates for different data can be stored in the electronic device. The scoring module 340 can determine the pre-stored data template corresponding to the data through the above-mentioned data tag. The pre-stored data template can include data structure templates, data format templates and other contents, and also include corresponding data scoring rules, such as 1 point deduction for data format errors, 0.5 points deduction for a data item not filled in, etc. After obtaining the above-mentioned pre-stored data templates, the scoring module 340 can compare the above-mentioned standard data with the above-mentioned pre-stored data templates, score the standard data, and obtain the above-mentioned data score. Among them, data comparison and scoring can be performed on all target data uploaded at one time, and data comparison and scoring can also be performed in groups.

[0161] After obtaining the data score, the scoring module 340 can determine whether the data score is higher than the first score threshold. If the data score is higher than the first score threshold, the data quality requirement is met and the data can be stored in the standard data group.

[0162] Optionally, the electronic device may further store multiple score thresholds, and the scoring module 340 may determine the quality level of the standard data by comparing with the above data score. For example, a second score threshold higher than the above first score threshold may be set, and the scoring module 340 may determine whether the data score is higher than the second score threshold. If the data score is higher than the second score threshold, the data quality requirement is met and the data quality level is excellent. If the data score is higher than the above first score threshold but not higher than the above second score threshold, the data quality requirement is met but the data quality level is good.

[0163] The grouping basis can be based on data quality level or data type, and there is no limitation here.

[0164] The evaluation of the data quality level by the scoring module 340 can organize the data according to the data quality situation, intuitively understand the data quality situation, and facilitate the subsequent data processing steps to refer to the data quality.

[0165] Optionally, the electronic device 300 also includes a reporting module 350 for generating a data detection report, wherein the data detection report includes the data detection time, data repetition rate, data qualification rate of the above-mentioned original data and / or the content of processing the above-mentioned target data according to the above-mentioned standardization rules, wherein the above-mentioned data repetition rate is the proportion of repeated data in the above-mentioned target data, and the above-mentioned data qualification rate is the proportion of standard data in the above-mentioned target data that meets the above-mentioned integrity check rules and the above-mentioned format rules.

[0166] The above-mentioned data detection report can clearly reflect the data quality situation, provide an effective reference for data acquisition and processing, and facilitate users to maintain data and improve data acquisition and data processing systems.

[0167] According to the specific implementation of the embodiment of this application, Figure 1 and Figure 2 The data processing method shown in the figure involves steps 101 to 104 and 201 to 208 which can be performed by Figure 3 The various modules in the electronic device 300 shown are executed. For example, Figure 1 Steps 101 to 104 in the above example can be respectively performed by Figure 3 The data is executed by the acquisition module 310, desensitization module 320 and data processing module 330 shown in FIG.

[0168] Through the electronic device 300 of the embodiment of the present application, the electronic device 300 can obtain the original data and the data labels carried by the above original data, desensitize the above original data, obtain the target data, and then obtain the standardization rules corresponding to the above data labels, the above standardization rules include integrity check rules and format rules, and then process the above target data according to the above standardization rules to obtain standard data that meets the above integrity check rules and the above format rules, which is convenient for standardizing data for subsequent data storage or processing, and can improve the efficiency and accuracy of data processing.

[0169] See also Figure 4 , Figure 4 is a schematic diagram of the structure of another electronic device disclosed in the embodiment of the present application. Figure 4 As shown, the electronic device 400 includes a processor 401 and a memory 402, wherein the electronic device 400 may also include a bus 403, the processor 401 and the memory 402 may be connected to each other via the bus 403, and the bus 403 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus 403 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The electronic device 400 may also include an input / output device 404, which may include a display screen, such as a liquid crystal display screen. The memory 402 is used to store one or more programs containing instructions; the processor 401 is used to call the instructions stored in the memory 402 to execute the above-mentioned Figure 1 and Figure 2 Some or all of the method steps mentioned in the examples.

[0170] It should be understood that in the embodiment of the present application, the processor 401 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0171] The input device 402 may include a touch panel, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 403 may include a display (LCD, etc.), a speaker, etc.

[0172] The memory 404 may include a read-only memory and a random access memory, and provide instructions and data to the processor 401. A portion of the memory 404 may also include a non-volatile random access memory. For example, the memory 404 may also store information about the device type.

[0173] Through the electronic device 400 of the embodiment of the present application, the electronic device 400 can obtain the original data and the data labels carried by the above original data, desensitize the above original data, obtain the target data, and then obtain the standardization rules corresponding to the above data labels, the above standardization rules include integrity check rules and format rules, and then process the above target data according to the above standardization rules to obtain standard data that meets the above integrity check rules and the above format rules, which is convenient for standardizing data for subsequent data storage or processing, and can improve the efficiency and accuracy of data processing.

[0174] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any data processing method recorded in the above method embodiments.

[0175] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0176] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the modules, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical or other forms.

[0177] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, magnetic disk or optical disk, etc., various media that can store program codes.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire original data and a data tag carried by the original data; the data tag is used to identify and determine the data type of the original data, and the data type refers to the data format of the original data; Performing desensitization processing on the original data to obtain target data; Obtaining standardization rules corresponding to the data tags, wherein the standardization rules include integrity check rules and format rules; Processing the target data according to the standardization rules to obtain standard data that meets the integrity check rules and the format rules; Obtain a pre-stored data template corresponding to the data tag, compare the standard data with the pre-stored data template, and obtain a data score; determine whether the data score is higher than a first score threshold, and if so, group and store the standard data.

2. The method according to claim 1, characterized in that In response to the standardization rule including the integrity check rule, processing the target data according to the standardization rule includes: Detect whether there are any vacancies in the target data, and if so, generate a report containing the vacancies, and output the report to prompt the completion of the vacancies.

3. The method according to claim 2, characterized in that The standardization rules further include duplicate checking rules, and processing the target data according to the standardization rules further includes: The similarity of the items in the target data is detected, and if the similarity is higher than a first threshold, the items are determined to be duplicate items, and the data of the duplicate items are merged or the data of one of the duplicate items is deleted.

4. The method according to claim 2, characterized in that: In response to the standardization rule including the format rule, processing the target data according to the standardization rule further includes: Check whether the project field in the target data matches the template field. If not, convert the unmatched project field into a target field that matches the template field according to the format rule.

5. The method according to claim 4, characterized in that The method further comprises: If it is detected that the project field that does not match the template field contains erroneous data, an error record is generated; Determine the item category of the item field, search for correction data corresponding to the item category and matching the item field in a correction database, and output prompt information containing the correction data.

6. The method according to any one of claims 1 to 5, characterized in that: After obtaining the standard data that meets the integrity check rule and the format rule, the method further includes: Generate a data detection report, the data detection report including the data detection time of the original data, the data repetition rate, the data qualification rate and / or the content of processing the target data according to the standardization rule, wherein the data repetition rate is the proportion of repeated data in the target data, and the data qualification rate is the proportion of standard data in the target data.

7. An electronic device, characterized in that: include: Scoring module, acquisition module, desensitization module and data processing module, among which: The acquisition module is used to acquire the original data and the data tag carried by the original data; the data tag is used to identify and determine the data type of the original data, and the data type refers to the data format of the original data; The desensitization module is used to perform desensitization processing on the original data to obtain target data; The acquisition module is also used to acquire the standardization rules corresponding to the data tags, wherein the standardization rules include integrity check rules and format rules; The data processing module is used to process the target data according to the standardization rules to obtain standard data that meets the integrity check rules and the format rules; The scoring module is used to obtain a pre-stored data template corresponding to the data tag, compare the standard data with the pre-stored data template, and obtain a data score; determine whether the data score is higher than a first score threshold, and if so, group and store the standard data.

8. An electronic device, characterized in that: The method comprises a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 6.

Images