Artificial intelligence system for data standardization

Abstract

The invention relates to the technical field of data standardization, in particular to an artificial intelligence system for data standardization, which comprises a data processing module, a data standardization module, a data confusion analysis module and a data division module. The data confusion analysis module uses a similarity comparison method to judge whether the data is removed as characters due to confusion of the characters and the data or whether the characters are reserved as the data when the redundant characters are removed, if the situation occurs, a similarity threshold value of the characters and the numbers is set based on a learning model, the confused data is re-divided, and the data confusion is completed. According to the method, the situation that valid data are deleted by mistake or invalid characters are reserved in the process of removing redundant characters is effectively avoided, the accuracy of data cleaning is ensured, the reserved data are really valuable data, and key information cannot be lost due to wrong cleaning operation.

Description

Technical Field

[0001] The present invention relates to the technical field of data standardization, and in particular to an artificial intelligence system for data standardization. Background Art

[0002] In the bank's information system, a huge amount of borrower information is stored. This information is crucial for business processes such as risk assessment, loan approval, and credit monitoring. Borrower information contains various types of data, such as personal identity information, financial status, credit record, loan history, etc.

[0003] However, the format and content of the above data are usually extremely complex and diverse, and there are inconsistencies and errors in data from different sources. If these data cannot be standardized, it will have a negative impact on the assessment of borrower information.

[0004] At the same time, in the process of data standardization, the data sources in the bank information system are mainly the information reserved by the borrower when handling the business, and the content entered into the bank system by the bank customer service after receiving the borrower's information;

[0005] However, there are some problems in this process. On the one hand, due to the different handwritings of different borrowers, numbers and similar characters are easily confused. On the other hand, when manually entering a large amount of data, data confusion may occur in the data table due to the lack of line breaks. If these two types of data confusion cannot be reasonably judged in time, it will affect the assessment of the borrower's credit. In view of this, we propose an artificial intelligence system for data standardization. Summary of the invention

[0006] The purpose of the present invention is to solve the problem of data confusion in the bank system during the credit assessment of the borrower, which causes the borrower's information data to be abnormal during the data standardization process;

[0007] To achieve the above-mentioned purpose, the present invention provides an artificial intelligence system for data standardization, which is capable of re-classifying abnormal borrower information data during data standardization, and includes a data processing module, a data standardization module, a data confusion analysis module and a data classification module;

[0008] The data processing module obtains the borrower information through the API interface protocol, unifies the borrower information through the format string and the string replacement method respectively, and the data standardization module is used to establish a data set corresponding to the borrower information, standardize the data set through the Z-score standardization method, and use the abnormal value definition method to determine the abnormal values ​​in the data set;

[0009] The data obfuscation analysis module uses a similarity comparison method to determine whether the abnormal value is caused by the string replacement method in the data processing module. If it is caused by the string replacement method, it means that the string replacement method confuses characters and numbers at this time, and the abnormal value is replaced again. If it is not caused by the string replacement method, it means that the abnormal value in the data standardization module is normal;

[0010] The data division module includes an unobfuscated data judgment unit and a re-division unit;

[0011] The unobfuscated data judgment unit is used to receive the number of abnormal values ​​determined by the data obfuscation analysis module if it is not caused by the string replacement method, and then judge whether the abnormal values ​​are adjacent in the data set based on the index position, and set a number threshold, and when the number of adjacent abnormal values ​​is greater than the number threshold, it indicates that there is data obfuscation in the data set;

[0012] The re-dividing unit is used to receive the data confusion signal, and re-divide the confused data by the abnormal value definition method in the data standardization module. After the re-dividing, the data is standardized again, and the standardized data is output to the staff.

[0013] The data processing module is used to send a data request and API key to the bank information system. The data request includes the borrower's account information, the borrower's credit report, date format information and the financial information submitted by the borrower. After receiving the request, the bank information system compares the API key with the legal key. If the API key = the legal key, it indicates that the verification is successful. At this time, the bank information system calls out the borrower's account information, the borrower's credit report, date format information and the financial information submitted by the borrower based on the borrower's identity information.

[0014] The steps of unifying date format information and numerical information by the data processing module through the format string and string replacement method are as follows:

[0015] Borrower information is divided into date format information and numerical information;

[0016] Receive date format information from the bank information system, group and understand the date strings in the date format information according to the definition of the format string through the format string, and then find the corresponding year, month and date parts according to the definition order of the format string to uniformly process the date format information;

[0017] Here is an example:

[0018] Assume that the date string is "2024 / 01 / 02" and the format string is "%Y / %m / %d". The format string recognizes that "2024" corresponds to "%Y", representing the year, "01" corresponds to "%m", representing the month, and "02" corresponds to "%d", representing the date, thus completing the mapping between the date string and the various parts of the date.

[0019] Parse the date format information in the order of the format string from left to right. For example, for the format string "%Y / %m / %d" and the date string "2024 / 01 / 02", first find the part corresponding to "%Y", that is, the year "2024", and get that "2024" in the date format information is the year. After finding the year, then find the corresponding month and date parts in the order of the format string, so as to process the date format information uniformly.

[0020] The steps to unify numeric information by string replacement are as follows:

[0021] Create replacement sets for unnecessary units and characters;

[0022] Starting from the beginning of the characters in the numeric information, compare them with the characters in the replacement set one by one.

[0023] If the characters in the numeric information = the characters in the replacement set, then remove the characters in the numeric information;

[0024] If the characters in the numeric information ≠ the characters in the replacement set, the characters in the numeric information will be retained.

[0025] The steps of standardizing the borrower information by the data standardization module using the Z-score standardization method are as follows:

[0026] Receiving the borrower's account information, the borrower's credit, the date format information and the financial information submitted by the borrower in the data processing module, sorting them in order according to the date format information, and establishing data sets corresponding to the date format information respectively;

[0027] Receive data collection: , where one of the data is , the calculation formula of Z-score is:

[0028] ;

[0029] in, is the standardized Z-score value, is the value of the original data point, n is a subscript used to distinguish the values ​​of different original data points. is the mean of the data set, The calculation formula is as follows:

[0030] ;

[0031] is the standard deviation of the data set, indicating the degree of dispersion of the data. The calculation formula is as follows:

[0032] .

[0033] The steps of the data standardization module to determine whether there are abnormal values ​​in the standardized data set are as follows:

[0034] Receive the Z-score value, mean and corresponding standard deviation of the standardized data set;

[0035] Abnormal numerical definition method:

[0036] Set abnormal value limits;

[0037] Calculate the mean In standard deviation The discrete degree is set as ;

[0038] like The data points fall on the mean Addition and subtraction times standard deviation In the range of , the interval is ;

[0039] like The mean of the data points Addition and subtraction times standard deviation In the range of , the interval is ;

[0040] On the contrary, data points outside this range are considered as outliers;

[0041] Abnormal value judgment:

[0042] Take interval ;

[0043] when When The value is an abnormal value.

[0044] The present invention fully considers the fact that different borrowers have different handwritings. Such differences can easily lead to confusion between numbers and similar characters. For example, the number "0" may be difficult to distinguish from the letter "O" due to the writing style, and the number "1" and the lowercase letter "l" may also be confused. Such confusion frequently occurs in the information filled in by borrowers with diverse handwritings, which will affect the assessment of the borrower's credit.

[0045] The steps of using the similarity comparison method in the data obfuscation analysis module are as follows:

[0046] Receiving the data corresponding to the abnormal value determined in the data standardization module and the replaced numeric characters in the data processing module, setting a similarity threshold based on the learning model, and determining whether the replaced characters are similar to the numeric values;

[0047] If they are similar, the replaced characters are converted into numerical values, and then the data standardization module is used to determine whether the numerical value is abnormal. Similarly, it can be determined whether the abnormal numerical value is caused by the similarity between the numerical value and the character;

[0048] Setting similarity threshold based on learning model The steps are as follows:

[0049] Receive known similar and dissimilar characters and the corresponding value ;

[0050] Similar and dissimilar and The same number of pixels at the same location: ;

[0051] in, and They are and exist Line The pixel value of the column, is the indicator function, if the pixel values ​​are the same then , otherwise ;

[0052] The formula for calculating character shape similarity is: ;

[0053] Then, compare the similarity distribution of similar and dissimilar corresponding characters and numerical values ​​to determine the similarity threshold ;

[0054] Receive data corresponding to abnormal values ​​in the data standardization module The characters corresponding to the data in the data processing module are replaced ;

[0055] Based on character shape similarity and similarity;

[0056] Receive the set similarity threshold ;

[0057] like ≥ , the output will convert the replaced characters into numerical values ​​or replace the numerical values ​​with characters;

[0058] like < , it means that the data processing module replaces the characters without confusion.

[0059] The present invention takes into account the risk of manually entering a large amount of data. When entering information into a data table, the staff may fail to perform a line break due to negligence or operational errors, which will cause data confusion. For example, data that should belong to different lines and represent different meanings are mistakenly connected together, causing originally independent borrower information fragments to be mixed with each other, which not only destroys the original structure of the data, but may also cause serious deviations in subsequent data processing links, which has a great negative impact on the accurate analysis of borrower information and credit assessment.

[0060] The steps of the unobfuscated data judging unit judging whether the unobfuscated data is adjacent to the corresponding data set based on the index position are as follows:

[0061] The data set of the data standardization module is sorted in the order of date format information. For multiple abnormal data, one abnormal data is selected as the index. , if the remaining abnormal data are respectively They are , ;

[0062] This indicates that the multiple abnormal data in the data set of the data standardization module (200) are adjacent data;

[0063] Set a quantity threshold, the quantity threshold is ≥ 2;

[0064] When the number of adjacent abnormal values ​​is greater than the number threshold, it indicates that there is data confusion in the data set;

[0065] The steps of re-dividing the abnormal data by the abnormal value definition method in the data standardization module are as follows:

[0066] Receiving the data obfuscation signal, obtaining the historical information of the corresponding borrower through the API interface protocol in the data processing module, and unifying the historical information respectively through the format string and string replacement method in the data processing module;

[0067] Calculate the boundary value of the historical data corresponding to the abnormal data by using the abnormal value boundary method of the data standardization module;

[0068] If the abnormal data is within the defined value, the abnormal data is reclassified, and then the abnormal data that has not been reclassified is classified in sequence;

[0069] After the division is completed, the data is standardized again through the data standardization module and output to the staff for the staff to evaluate the borrower.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] 1. In this artificial intelligence system for data standardization, a borrower information set is established through a data standardization module, and then the data set is standardized using a Z-score standardization method. The Z-score standardization method can convert borrower information data of different magnitudes and dimensions into a comparable standard form, so that when conducting credit assessment, the relationship between various features can be more accurately analyzed and measured. Prior to this, the borrower information is unified through a data processing module, redundant characters are removed, and the date format is unified, which can eliminate data format differences caused by different entry methods or sources, and help reduce errors introduced by confusing data formats.

[0072] 2. In this artificial intelligence system for data standardization, after the data set is standardized, the data confusion analysis module uses a similarity comparison method to determine whether the data is removed as characters due to confusion between characters and data when removing redundant characters, or whether the characters are retained as data. If the above situation occurs, the similarity threshold between characters and numbers is set based on the learning model, and the confused data is re-divided, effectively avoiding the accidental deletion of valid data or retention of invalid characters in the process of removing redundant characters, ensuring the accuracy of data cleaning, so that the retained data are truly valuable data, and key information will not be lost due to incorrect cleaning operations.

[0073] 3. In this artificial intelligence system for data standardization, if the data confusion analysis module uses the similarity comparison method to determine that there is confusion between characters and data, the unconfused data judgment unit determines whether the abnormal data is adjacent data. If it is adjacent data, it means that there may be a possibility of adjacent data mixing at this time, avoiding blind processing of the entire data set and focusing on solving problems that may arise from the mixing of adjacent data. At this time, the redivision unit redivides the abnormal data through the abnormal value definition method in the data standardization module. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is the overall module principle diagram of the present invention.

[0075] The meaning of each number in the figure is:

[0076] 100, data processing module; 200, data standardization module; 300, data confusion analysis module; 400, data partitioning module; 410, unobfuscated data judgment unit; 420, re-partitioning unit. DETAILED DESCRIPTION

[0077] The following will be combined with the accompanying drawings in the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] An artificial intelligence system for data standardization includes a data processing module 100, a data standardization module 200, a data confusion analysis module 300 and a data partitioning module 400;

[0079] The data processing module 100 obtains the borrower information through the API interface protocol, and unifies the borrower information through the format string and the string replacement method. The API interface follows strict data transmission specifications, which can ensure the accuracy of data during the transmission process and reduce data errors caused by manual intervention or non-standard transmission. In the past, for the problem of inconsistent data formats, some practices may be simple manual visual recognition and manual modification, which is extremely inefficient and difficult to ensure the consistency of unified standards. Some may use more complex scripts for processing, but they are not very versatile. Once there are new changes in the data format, a large amount of code needs to be rewritten for adjustment. For example, faced with different formats of borrower names, some are all capitalized, some have the first letter capitalized and the rest are lowercase, and some are interspersed with spaces. If you modify them one by one manually, the workload is huge, and traditional fixed-rule scripts are difficult to cope with the changing actual formats.

[0080] The data processing module 100 is used to send a data request and an API key to a bank information system. The data request includes the borrower's account information, the borrower's credit report, date format information, and financial information submitted by the borrower. After receiving the request, the bank information system compares the API key with the legal key. If the API key = the legal key, it indicates that the verification is successful. At this time, the bank information system retrieves the borrower's account information, the borrower's credit report, date format information, and financial information submitted by the borrower based on the borrower's identity information.

[0081] The steps of the data processing module 100 unifying the date format information and the numerical information through the format string and the string replacement method are as follows:

[0082] Borrower information is divided into date format information and numerical information;

[0083] Receive date format information from the bank information system, group and understand the date strings in the date format information according to the definition of the format string through the format string, and then find the corresponding year, month and date parts according to the definition order of the format string to uniformly process the date format information;

[0084] Here is an example:

[0085] Assume that the date string is "2024 / 01 / 02" and the format string is "%Y / %m / %d". The format string recognizes that "2024" corresponds to "%Y", representing the year, "01" corresponds to "%m", representing the month, and "02" corresponds to "%d", representing the date, thus completing the mapping between the date string and the various parts of the date.

[0086] Parse the date format information in the order of the format string from left to right. For example, for the format string "%Y / %m / %d" and the date string "2024 / 01 / 02", first find the part corresponding to "%Y", that is, the year "2024", and get that "2024" in the date format information is the year. After finding the year, then find the corresponding month and date parts in the order of the format string, so as to process the date format information uniformly.

[0087] The steps to unify numeric information by string replacement are as follows:

[0088] Create replacement sets for unnecessary units and characters;

[0089] Starting from the beginning of the characters in the numeric information, compare them with the characters in the replacement set one by one.

[0090] If the characters in the numeric information = the characters in the replacement set, then remove the characters in the numeric information;

[0091] If the characters in the numeric information ≠ the characters in the replacement set, the characters in the numeric information will be retained.

[0092] The data standardization module 200 is used to establish a data set corresponding to the borrower information, standardize the data set through the Z-score standardization method, and use the abnormal value definition method to determine the abnormal values ​​in the data set. The format string provides a template for formatting data, which can flexibly deal with borrower information in various formats. By defining the format template, such as stipulating that the format of the name is "capital first letter, lowercase rest, separated by spaces", it can adapt to the name data of different initial formats for unified conversion. The string replacement method can accurately replace some specific characters and symbols, such as uniformly replacing " / " in the date format with "-", etc. It is simple to operate and can quickly adapt to the subtle differences of different data. It has strong versatility and does not require writing complex customized code for each specific situation.

[0093] Regardless of the original range and magnitude of the borrower information data, Z-score standardization can effectively process it to meet the general standardization requirements. It does not rely on specific data range settings or subjective experience values. As long as the data meets certain conditions for calculable mean and standard deviation, usually numerical data basically meets them and plays a good role. It is widely applicable to various scenarios involving borrower information analysis, modeling, etc., and is conducive to the integration and comparison of different data sets.

[0094] The steps of the data standardization module 200 to standardize the borrower information by using the Z-score standardization method are as follows:

[0095] Receive the borrower account information, the borrower credit, the date format information and the financial information submitted by the borrower in the data processing module 100, sort them in order according to the date format information, and respectively establish data sets corresponding to the date format information;

[0096] Receive data collection: , where one of the data is , the calculation formula of Z-score is:

[0097] ;

[0098] in, is the standardized Z-score value, is the value of the original data point, is the mean of the data set, The calculation formula is as follows:

[0099] ;

[0100] is the standard deviation of the data set, indicating the degree of dispersion of the data. The calculation formula is as follows:

[0101] .

[0102] The steps of the data standardization module 200 to determine whether there are abnormal values ​​in the standardized data set are as follows:

[0103] Receive the Z-score value, mean and corresponding standard deviation of the standardized data set;

[0104] Abnormal numerical definition method:

[0105] Set abnormal value limits;

[0106] Calculate the mean In standard deviation The discrete degree is set to ;

[0107] like The data points fall on the mean Addition and subtraction times standard deviation In the range of , the interval is ;

[0108] like The mean of the data points Addition and subtraction times standard deviation In the range of , the interval is ;

[0109] On the contrary, data points outside this range are considered as outliers;

[0110] Abnormal value judgment:

[0111] Take interval ;

[0112] when When The value is an abnormal value.

[0113] Traditional data anomaly processing methods often focus on simply correcting the abnormal values ​​themselves or directly marking and removing them, and rarely go deep into tracing the specific causes of the anomalies, especially analyzing whether they are caused by specific data processing steps such as string replacement.

[0114] For example, when an abnormality is found in the borrower's income value, it is simply judged to be beyond the conventional setting range, and then the value is either manually adjusted or corrected according to fixed rules, without exploring whether the data is confused due to string replacement operations in the previous pre-processing stages such as data format unification. As a result, the problem cannot be solved from the root, and similar abnormalities may occur repeatedly in the future;

[0115] The data obfuscation analysis module 300 uses a similarity comparison method to determine whether the abnormal value is caused by the string replacement method in the data processing module 100. If it is caused by the string replacement method, it means that the string replacement method confuses characters and numbers at this time and replaces the abnormal value again. If it is not caused by the string replacement method, it means that the abnormal value in the data standardization module 200 is normal. By using the similarity comparison method to specifically determine whether the abnormal value is caused by the string replacement method, the source of the abnormality can be accurately traced back;

[0116] For example, if the telephone number field in the borrower's information has an abnormal format, such as characters mixed into the original pure numeric number, by comparing the data features and formats before and after the replacement through the similarity comparison method, it can be determined whether the abnormality is caused by the misreplacement of certain similar characters during the string replacement operation, such as the number "0" being mistakenly replaced with the letter "O". This kind of confusion frequently occurs in the information filled out by borrowers with diverse handwriting, which will affect the assessment of the borrower's credit.

[0117] The steps of the data obfuscation analysis module 300 using the similarity comparison method are as follows:

[0118] Receive the data corresponding to the abnormal value determined in the data standardization module 200 and the replaced numeric characters in the data processing module 100, set a similarity threshold based on the learning model, and determine whether the replaced characters are similar to the numeric values;

[0119] If they are similar, the replaced characters are converted into numerical values, and then the data standardization module 200 determines whether the numerical value is abnormal. Similarly, it can be determined whether the abnormal numerical value is caused by the similarity between the numerical value and the character;

[0120] Setting similarity threshold based on learning model The steps are as follows:

[0121] Receive known similar and dissimilar characters and the corresponding value ;

[0122] Similar and dissimilar and The same number of pixels at the same location: ;

[0123] in, and They are and exist Line The pixel value of the column, is the indicator function, if the pixel values ​​are the same then , otherwise ;

[0124] The formula for calculating character shape similarity is: ;

[0125] Then, compare the similarity distribution of similar and dissimilar corresponding characters and numerical values ​​to determine the similarity threshold ;

[0126] The data corresponding to the abnormal value determined in the receiving data standardization module 200 is The characters corresponding to the data in the data processing module 100 are replaced ;

[0127] Based on character shape similarity and similarity;

[0128] Receive the set similarity threshold ;

[0129] like ≥ , the output will convert the replaced characters into numerical values ​​or replace the numerical values ​​with characters;

[0130] like < , it means that the characters replaced by the data processing module 100 are not confused.

[0131] The present invention takes into account the risk of manually entering a large amount of data. When entering information into a data table, the staff may fail to perform a line break due to negligence or operational errors, which will cause data confusion. For example, data that should belong to different lines and represent different meanings are mistakenly connected together, causing originally independent borrower information fragments to be mixed with each other, which not only destroys the original structure of the data, but may also cause serious deviations in subsequent data processing links, which has a great negative impact on the accurate analysis of borrower information and credit assessment.

[0132] The steps of the unobfuscated data judging unit 410 judging whether the unobfuscated data is adjacent to the corresponding data set based on the index position are as follows:

[0133] The data set of the data standardization module 200 is sorted in the order of date format information. For multiple abnormal data, one abnormal data is selected as the index. , if the remaining abnormal data are respectively They are , ;

[0134] This indicates that the multiple abnormal data in the data set of the data standardization module (200) are adjacent data;

[0135] Set a quantity threshold, the quantity threshold is ≥ 2;

[0136] When the number of adjacent abnormal values ​​is greater than the number threshold, it indicates that there is data confusion in the data set;

[0137] By judging whether the abnormal values ​​are adjacent in the data set based on the index position and considering the number of adjacent abnormal values, anomalies can be discovered from the perspective of the overall layout and correlation of the data;

[0138] For example, in a data set related to borrower credit scores, if a certain indicator of multiple consecutive borrowers, such as debt ratio, is abnormal, and these abnormal values ​​are arranged adjacent to each other in the data set, it may imply that there is some systematic data confusion or error source. Perhaps the data collection equipment fails, or a specific business process link has an error, which causes the problem with this piece of data. This kind of analysis from the perspective of location association helps to dig deeper into the deep-seated problems hidden behind the data, avoids just superficially dealing with a single abnormal value, and improves the comprehensiveness of data quality control.

[0139] The steps of re-dividing the abnormal data by the abnormal value definition method in the data standardization module 200 are as follows:

[0140] Receive the data obfuscation signal, obtain the historical information of the corresponding borrower through the API interface protocol in the data processing module 100, and unify the historical information through the format string and string replacement method in the data processing module 100;

[0141] Calculate the boundary value of the historical data corresponding to the abnormal data by using the abnormal value boundary method of the data standardization module 200;

[0142] If the abnormal data is within the defined value, the abnormal data is reclassified, and then the abnormal data that has not been reclassified is classified in sequence;

[0143] After the division is completed, the data is standardized again by the data standardization module 200 and output to the staff for the staff to evaluate the borrower;

[0144] When receiving the data confusion signal, the existing abnormal value definition method in the data standardization module 200 is used to re-divide the confused data, so as to trace back to the link where the data problem occurs, sort out the confused data according to scientific and reasonable definition standards, and re-distinguish it according to reasonable logic and business rules, find out the data part that was originally wrongly divided or disturbed, and realize accurate repair;

[0145] For example, for data related to the borrower's repayment ability, if income, expenditure and other data are mixed together due to certain external factors, the abnormal value definition method can be used to redefine the reasonable numerical range of each part, divide them correctly, restore the accuracy and availability of the data, and avoid the information loss caused by simply discarding the data.

[0146] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An artificial intelligence system for data standardization, characterized in that: It comprises a data processing module (100), a data standardization module (200), a data confusion analysis module (300) and a data partitioning module (400); The data processing module (100) obtains borrower information through an API interface protocol, unifies the borrower information through a format string and a string replacement method, and the data standardization module (200) is used to establish a data set corresponding to the borrower information, standardize the data set through a Z-score standardization method, and use an abnormal value definition method to determine abnormal values ​​in the data set; The data confusion analysis module (300) uses a similarity comparison method to determine whether the abnormal value is caused by the string replacement method in the data processing module (100). If it is caused by the string replacement method, it means that the string replacement method confuses characters and numbers at this time, and the abnormal value is replaced again. If it is not caused by the string replacement method, it means that the abnormal value in the data standardization module (200) is normal; The data partitioning module (400) comprises an unobfuscated data determination unit (410) and a re-partitioning unit (420); The unobfuscated data judgment unit (410) is used to receive the number of abnormal values ​​determined by the data obfuscation analysis module (300) if it is not caused by the string replacement method, and then judge whether the abnormal values ​​are adjacent in the data set based on the index position, and set a number threshold, and when the number of adjacent abnormal values ​​is greater than the number threshold, it indicates that there is data obfuscation in the data set; The re-dividing unit (420) is used to receive the data confusion signal and re-divide the confused data through the abnormal value definition method in the data standardization module (200); after the re-dividing, the data is standardized again and the standardized data is output to the staff.

2. The artificial intelligence system for data standardization according to claim 1, characterized in that: The data processing module (100) is used to send a data request and an API key to a bank information system. The data request includes the borrower's account information, the borrower's credit report, date format information, and financial information submitted by the borrower. After receiving the request, the bank information system compares the API key with the legal key. If the API key = the legal key, it indicates that the verification is successful. At this time, the bank information system retrieves the borrower's account information, the borrower's credit report, date format information, and financial information submitted by the borrower based on the borrower's identity information.

3. The artificial intelligence system for data standardization according to claim 1, characterized in that: The steps of the data processing module (100) unifying date format information and numerical information through format strings and string replacement methods are as follows: Borrower information is divided into date format information and numerical information; Receive the date format information from the bank information system, and convert the date string in the date format information into a format string; Group and understand as defined by the format string; The format string looks for the corresponding year, month, and day parts in the order they are defined; Unify the date format information; The steps to unify numeric information by string replacement are as follows: Create replacement sets for unnecessary units and characters; Starting from the beginning of the characters in the numeric information, compare them with the characters in the replacement set one by one. If the characters in the numeric information = the characters in the replacement set, then remove the characters in the numeric information; If the characters in the numeric information ≠ the characters in the replacement set, the characters in the numeric information will be retained.

4. The artificial intelligence system for data standardization according to claim 1, characterized in that: The steps of the data standardization module (200) for standardizing the borrower information by using the Z-score standardization method are as follows: Receiving the borrower's account information, the borrower's credit, date format information and the financial information submitted by the borrower in the data processing module (100), sorting them in order according to the date format information, and establishing data sets corresponding to the date format information respectively; Receive data collection: , where one of the data is , the calculation formula of Z-score is: ； in, is the standardized Z-score value, is the value of the original data point, is the mean of the data set, The calculation formula is as follows: ； is the standard deviation of the data set, indicating the degree of dispersion of the data. The calculation formula is as follows: 。 5. The artificial intelligence system for data standardization according to claim 1, characterized in that: The steps of the data standardization module (200) for determining whether there are abnormal values ​​in the standardized data set are as follows: Receive the Z-score value, mean and corresponding standard deviation of the standardized data set; Abnormal numerical definition method: Set abnormal value limits; Calculate the mean In standard deviation The discrete degree is set to ; like The data points fall on the mean Addition and subtraction times standard deviation In the range of , the interval is , like The mean of the data points Addition and subtraction times standard deviation In the range of , the interval is , On the contrary, data points outside this range are considered as outliers; Abnormal value judgment: Take interval ; when When The value is an abnormal value.

6. The artificial intelligence system for data standardization according to claim 1, characterized in that: The steps of the data obfuscation analysis module (300) using the similarity comparison method are as follows: Receiving the data corresponding to the abnormal numerical value determined in the data standardization module (200) and the replaced numerical characters in the data processing module (100), setting a similarity threshold based on the learning model, and determining whether the replaced characters are similar to the numerical values; If they are similar, the replaced characters are converted into numerical values, and then the data standardization module (200) is used to determine whether the numerical value is abnormal. Similarly, it can be determined whether the abnormal numerical value is caused by the similarity between the numerical value and the characters; Setting similarity threshold based on learning model The steps are as follows: Receive known similar and dissimilar characters and the corresponding value ; Similar and dissimilar and The same number of pixels at the same location: ; in, and They are and exist Line The pixel value of the column, is the indicator function, if the pixel values ​​are the same then , otherwise ; The formula for calculating character shape similarity is: ; Compare the similarity distribution of similar and dissimilar characters and numerical values ​​to determine the similarity threshold ; The data corresponding to the abnormal value determined in the receiving data standardization module (200) The characters corresponding to the data in the data processing module (100) are replaced ; Based on character shape similarity and similarity; Receive the set similarity threshold ; like ≥ , the output will convert the replaced characters into numerical values ​​or replace the numerical values ​​with characters; like < , it means that the data processing module (100) replaces characters without confusion.

7. The artificial intelligence system for data standardization according to claim 1, characterized in that: The steps of the unobfuscated data determination unit (410) determining whether an abnormal value is adjacent to its corresponding data set based on the index position are as follows: The data standardization module (200) sorts the data set according to the order of date format information; For multiple abnormal data, select one abnormal data as the index ; If the remaining abnormal data are respectively They are , ; This indicates that the multiple abnormal data in the data set of the data standardization module (200) are adjacent data; Set a quantity threshold, the quantity threshold is ≥ 2; When the number of adjacent abnormal values ​​is greater than the number threshold, it indicates that there is data confusion in the data set.

8. The artificial intelligence system for data standardization according to claim 1, characterized in that: The steps of re-dividing the abnormal data by the abnormal value definition method in the data standardization module (200) are as follows: Receiving a data obfuscation signal, obtaining historical information of the corresponding borrower through an API interface protocol in a data processing module (100), and unifying the historical information through a format string and a string replacement method in the data processing module (100); Calculating the boundary value of the historical data corresponding to the abnormal data by using the abnormal value boundary method of the data standardization module (200); If the abnormal data is within the defined value, the abnormal data is reclassified, and then the abnormal data that has not been reclassified is classified in sequence; After the division is completed, the data is standardized again through the data standardization module (200) and output to the staff for the staff to evaluate the borrower.

Images