Abnormal data detection method
By generating and filtering the character and string collections in the historical data set of the data platform, consistency matching verification is performed, the problem of inefficiency of traditional manual detection methods is solved, the accuracy and efficiency of abnormal data detection is improved, and the data is ensured. Reliability and security of processing.
Patent Information
- Application Number
- CN202410956979.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-08
AI Technical Summary
In big data platforms, traditional artificial anomaly data detection methods are inefficient and expensive, making it difficult to capture subtle changes and patterns in the data, making it difficult to identify potential anomalies or fraudulent behaviors, and may even lead to failure of the big data platform.
By obtaining historical data sets, a deduplication set of characters and strings is generated, and a target string set is intelligently filtered based on these sets, and consistency matching checks are performed to identify exception data. This method uses the quantitative relationship between characters and strings to filter out the key target string sets used for exception detection, which improves the accuracy and efficiency of detection.
It improves the accuracy and efficiency of abnormal data detection, ensures data quality and security, significantly reduces labor costs and calculation costs, and ensures the reliability and effectiveness of data processing.
Smart Images

Figure CN118916706A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer technology, and specifically relates to an abnormal data detection method, an abnormal data detection device, a computer-readable medium, an electronic device, and a computer program product. Background Art
[0002] In the massive data of the big data platform, various forms of data are stored, such as system serial numbers, status codes, mobile phone numbers, ID card numbers, etc. Due to the huge amount of data, the diversity of data forms and the high frequency of data updates, data anomaly detection becomes extremely complex. Traditional manual detection methods are not only inefficient but also costly. Especially in the case of real-time monitoring and analysis throughout the day, manual methods are difficult to capture subtle changes and patterns in the data. These changes and patterns may indicate potential anomalies or fraudulent behavior, and in serious cases, they may lead to problems such as failure of the big data platform. Summary of the invention
[0003] The present application provides an abnormal data detection method, an abnormal data detection device, a computer-readable medium, an electronic device, and a computer program product, the purpose of which is to improve the detection efficiency and detection success rate of abnormal data detection.
[0004] According to one aspect of an embodiment of the present application, a method for detecting abnormal data is provided, the method comprising: obtaining a historical data set, the historical data set comprising multiple data items, the data items being a character sequence consisting of multiple characters; generating a character set and a string set corresponding to each sequence position in the character sequence based on the historical data set, the character set being a de-duplicated set consisting of characters located at one sequence position, and the string set being a de-duplicated set consisting of strings located at at least two sequence positions; screening a target string set for abnormal data detection based on the number of characters in the character set and the number of strings in the string set; performing a consistency matching check between the data to be detected and the target string set, and determining that the data to be detected is abnormal data when the check fails.
[0005] According to one aspect of an embodiment of the present application, there is provided an abnormal data detection device, the device comprising:
[0006] An acquisition module is configured to acquire a historical data set, wherein the historical data set includes a plurality of data items, and the data items are character sequences composed of a plurality of characters;
[0007] a generating module configured to generate, based on the historical data set, a character set and a string set corresponding to each sequence position in the character sequence, wherein the character set is a duplicate-free set consisting of characters located at one sequence position, and the string set is a duplicate-free set consisting of strings located at at least two sequence positions;
[0008] A screening module is configured to screen a target character string set for abnormal data detection according to the number of characters in the character set and the number of character strings in the character string set;
[0009] The verification module is configured to perform consistency matching verification on the data to be detected and the target character string set, and determine that the data to be detected is abnormal data when the verification fails.
[0010] In some embodiments of the present application, based on the above technical solution, the generation module includes:
[0011] A splitting module is configured to split each data item in the historical data set into a sequence according to a single character and combine them to obtain a character matrix, wherein a character row in the character matrix corresponds to a data item, and a character column in the character matrix corresponds to a sequence position;
[0012] A combining module, configured to remove duplicates from all characters in a character column to form a character set corresponding to a position in the sequence;
[0013] The concatenation module is configured to concatenate characters on two character columns according to the arrangement order of the sequence positions to form character strings corresponding to each character row, and to remove duplicates from the character strings to form a character string set corresponding to the two sequence positions.
[0014] In some embodiments of the present application, based on the above technical solution, the character string includes an adjacent character string and a non-adjacent character string, the adjacent character string is composed of the concatenation of two characters whose sequence positions are adjacent, and the non-adjacent character string is composed of the concatenation of two characters whose sequence positions are not adjacent.
[0015] In some embodiments of the present application, based on the above technical solution, the screening module includes:
[0016] a quantity determination module configured to determine the number of character combinations corresponding to two sequence positions according to the number of characters in the character set, the number of character combinations being the maximum number of character combinations generated by freely combining the characters at the two sequence positions;
[0017] The numerical comparison module is configured to screen a target character string set for abnormal data detection according to a numerical relationship between the number of character combinations and the number of character strings.
[0018] In some embodiments of the present application, based on the above technical solution, the numerical comparison module includes:
[0019] A first comparison module is configured to, when the number of character combinations is less than a preset first number threshold, determine a character string set corresponding to the number of character strings as a target character string set for abnormal data detection if the number of character combinations is greater than the number of character strings;
[0020] The second comparison module is configured to perform a correlation check on two sequence positions corresponding to the number of character combinations when the number of character combinations is greater than the first number threshold, and determine the string set corresponding to the number of character strings as a target string set for abnormal data detection when the check is successful.
[0021] In some embodiments of the present application, based on the above technical solution, the second comparison module is also configured to: obtain a ratio threshold corresponding to the two sequence positions, and the ratio threshold is less than 1; if the product of the number of character combinations and the ratio threshold is greater than the number of character strings, then determine that the two sequence positions corresponding to the number of character combinations are associated.
[0022] In some embodiments of the present application, based on the above technical solution, the second comparison module is further configured to: count the actual occurrence frequency and theoretical occurrence frequency of any pair of character combinations at the two sequence positions corresponding to the number of character combinations; determine the reliability coefficient of the correlation between the two sequence positions based on the actual occurrence frequency and the theoretical occurrence frequency; if the reliability coefficient is greater than a preset reliability threshold, it is determined that the two sequence positions corresponding to the number of character combinations are correlated.
[0023] In some embodiments of the present application, based on the above technical solution, the reliability coefficient is calculated according to the following formula:
[0024]
[0025] Among them, x 2 is the reliability coefficient, y1 is the actual occurrence frequency of the character combination, and y2 is the theoretical occurrence frequency of the character combination.
[0026] In some embodiments of the present application, based on the above technical solution, the second comparison module is further configured to: when the number of character combinations is less than a preset second number threshold, obtain a ratio threshold corresponding to the two sequence positions, and the ratio threshold is less than 1; if the product of the number of character combinations and the ratio threshold is greater than the number of character strings, determine that the two sequence positions corresponding to the number of character combinations are associated; when the number of character combinations is greater than the second number threshold, count the actual occurrence frequency and theoretical occurrence frequency of any pair of character combinations at the two sequence positions corresponding to the number of character combinations; determine the reliability coefficient of the association between the two sequence positions based on the actual occurrence frequency and the theoretical occurrence frequency; if the reliability coefficient is greater than the preset reliability threshold, determine that the two sequence positions corresponding to the number of character combinations are associated.
[0027] In some embodiments of the present application, based on the above technical solution, the device further includes:
[0028] The merging module is configured to merge any two target character string sets when the two target character string sets correspond to a same sequence position.
[0029] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the abnormal data detection method in the above technical solution is implemented.
[0030] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the abnormal data detection method in the above technical solution.
[0031] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the abnormal data detection method in the above technical solution is implemented.
[0032] In the technical solution provided in the embodiment of the present application, by efficiently utilizing the historical data set, a deduplication set containing all characters and strings is first constructed, and then based on the number of characters and strings in these sets, a key target string set for anomaly detection is intelligently screened. This method not only improves the accuracy of abnormal data detection, but also can quickly identify and determine anomalies in the data to be detected through consistency matching verification, thereby providing significant benefits in terms of data quality and security, and ensuring the reliability and effectiveness of data processing.
[0033] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1 The exemplary system architecture block diagram applying the technical solution of the present application is schematically shown.
[0036] Figure 2 A flow chart of an abnormal data detection method in one embodiment of the present application is shown.
[0037] Figure 3 A flowchart of generating a character set and a string set in one embodiment of the present application is shown.
[0038] Figure 4 A character matrix of an embodiment of the present application in an application scenario is shown.
[0039] Figure 5 The character set corresponding to each character column in an application scenario of an embodiment of the present application is shown.
[0040] Figure 6 The figure shows a combination of character strings used to concatenate character strings in an application scenario according to an embodiment of the present application.
[0041] Figure 7 A frequency matrix for counting occurrence frequencies in an application scenario according to an embodiment of the present application is shown.
[0042] Figure 8 The frequency statistics result of the embodiment of the present application in an application scenario is shown.
[0043] Fig. 9 The statistical result of the occurrence frequency of the character combination in an application scenario of the embodiment of the present application is shown.
[0044] Fig.10 It shows a plurality of target character string sets formed by segmenting a character sequence in an application scenario of an embodiment of the present application.
[0045] Fig.11 A schematic diagram showing a consistency match check between the data to be detected and the target string set in an application scenario according to an embodiment of the present application is shown.
[0046] Fig.12 The structural block diagram of the abnormal data detection device provided in an embodiment of the present application is schematically shown.
[0047] Fig.13 The structure block diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application is schematically shown. DETAILED DESCRIPTION
[0048] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of example embodiments to those skilled in the art.
[0049] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.
[0050] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0051] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0052] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0053] In the related art of this application, the Apriori algorithm uses frequent item sets to generate association rules, which is designed for databases containing transactions. With the help of these association rules, it can determine the connection strength between two objects. Assume that there is a data set containing various transactions as shown in Table 1 below. Frequent item sets need to be found from the data set and association rules need to be generated using the Apriori algorithm.
[0054] Table 1
[0055]
[0056]
[0057] Define the minimum support as 2, and the execution process of the Apriori algorithm includes the following steps.
[0058] Step 1: Calculate C1 and L1.
[0059] In the first step, create Table 2 as shown below, which contains the support count SUPORT_COUNT (the frequency of each item set in the dataset) of each item set in the given dataset. This table is called candidate set C1.
[0060] Table 2
[0061] ITEMSET SUPORT_COUNT A 6 B 7 C 6 D 2 E 1
[0062] According to Table 2, all item sets whose support counts are greater than the minimum support (2) in the candidate set C1 are taken out to obtain the frequent item set L1 shown in Table 3 below.
[0063] Table 3
[0064] ITEMSET SUPORT_COUNT A 6 B 7 C 6 D 2
[0065] As shown in Table 3, since the support of all itemsets except E is greater than or equal to the minimum support, the E itemset will be deleted.
[0066] Step 2: Generate C2 and L2.
[0067] Generate candidate set C2 using frequent item set L1. In candidate set C2, create item set pairs of frequent item set L1 in the form of subsets. After creating the subsets, look up the support count from the main transaction table of the dataset again, that is, the number of times these pairs appear together in a given dataset. Based on the above process, we can get candidate set C2 as shown in Table 4 below.
[0068] Table 4
[0069] ITEMSET SUPORT_COUNT A,B 4 A,C 4 A,D 1 B,C 4 B,D 2 C,D 0
[0070] Compare each support count in the candidate set C2 with the minimum support. After comparison, the item sets with less support counts will be removed from the candidate set C2, and the frequent item set L2 shown in Table 5 below will be obtained.
[0071] Table 5
[0072] ITEMSET SUPORT_COUNT A,B 4 A,C 4 B,C 4 B,D 2
[0073] Step 3: Generate C3 and L3.
[0074] The candidate set C3 is formed by combining subsets of the three item sets, and the support counts are calculated from the dataset, resulting in Table 6 below.
[0075] Table 6
[0076] ITEMSET SUPORT_COUNT A, B, C 2 B,C,D 0 A,C,D 0 A, B, D 0
[0077] Based on the candidate set C3, we continue to create the frequent item set L3. As can be seen from Table 6, there is only one item set combination whose support is equal to the minimum support. Therefore, the frequent item set L3 will have only one combination, namely {A, B, C}.
[0078] From the above calculation process of the Apriori algorithm, we can see that the longest frequent itemset can be calculated by the Apriori algorithm. However, this calculation process ignores the infrequent itemsets in the data.
[0079] In the abnormal data identification scenario, since all regular item sets of the data need to be mined, infrequent item sets need to be included, that is, the minimum support needs to be set to 1. The Apriori algorithm itself has a large amount of calculation and database query. Each round of frequent item set mining requires re-traversal of the database and calculation. If infrequent item sets are to be mined, this will lead to a large amount of database traversal and set calculation, which may even increase exponentially in the worst case.
[0080] In addition, the rule mining of the Apriori algorithm is a completely free combination, which will disrupt the position of the data columns for combination, thereby losing the position information, leading to problems such as low efficiency in abnormal data recognition and low recognition success rate.
[0081] In other words, the calculation mode of the Apriori algorithm is to perform pruning and calculations in each round based on support and confidence. Because the support needs to be calculated, the support must be recalculated in each round, and the results calculated in the previous round cannot be reused, resulting in a large amount of repeated data traversal and calculation. As a data mining algorithm, Apriori is suitable for calculating the longest frequent item set scenario. It uses free combination of data, but in the scenario of abnormal data identification, its calculation method will cause mismatch and lose the location information of the data.
[0082] In response to the problems existing in the above-mentioned related technologies, this application starts from the calculation of combination constraints, and no longer uses the minimum support method of the Apriori algorithm. It only needs to traverse the data set once, and at the same time calculate the deduplication set of all column values and the deduplication set of two-column combinations, and reuse the deduplication set of two-column combinations to derive subsequent combination constraint rules, avoiding the need to traverse the database again and repeat calculations every time frequent item sets are mined. Moreover, this application is designed directly for the scenario of abnormal data identification, and the combination derivation and rule mining are performed strictly in accordance with the column alignment method, which will not cause mismatches. It not only makes full use of the information of data location, but also allows free combination of different columns under the premise of column alignment.
[0083] Taking the deposit core system of the banking industry as an example, this application can automatically perform anomaly detection on all data on the current deposit big data platform. As a monitoring of business data, it can automatically, efficiently and accurately detect the current business data with anomalies. At present, the deposit big data platform already has more than 100 million data, but whether the data is correct and whether there are anomalies is very costly to identify manually. This application learns the composition rules of data from massive data in an unsupervised manner, and automatically identifies anomalies of the data based on this rule, which can greatly save labor costs and reduce computing costs.
[0084] Figure 1 The exemplary system architecture block diagram applying the technical solution of the present application is schematically shown.
[0085] like Figure 1 As shown, the system architecture 100 may include a terminal device 110, a network 120 and a server 130. The terminal device 110 may include various electronic devices such as smart phones, tablet computers, laptops, desktop computers, smart speakers, smart wearable devices, smart vehicle-mounted devices, and smart payment terminals. The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The network 120 may be a communication medium of various connection types that can provide a communication link between the terminal device 110 and the server 130, such as a wired communication link or a wireless communication link.
[0086] According to the implementation requirements, the system architecture in the embodiment of the present application can have any number of terminal devices, networks and servers. For example, the server 130 can be a server group composed of multiple server devices. In addition, the technical solution provided in the embodiment of the present application can be applied to the terminal device 110, can also be applied to the server 130, or can be implemented by the terminal device 110 and the server 130 together, and the present application does not make any special restrictions on this.
[0087] Figure 2 A flowchart of an abnormal data detection method in an embodiment of the present application is shown. The abnormal data detection method can be performed by Figure 1 The terminal device or server shown in the figure is executed, and the embodiment of the present application takes the abnormal data detection method executed by the terminal device as an example to illustrate. Figure 2 As shown, the abnormal data detection method includes the following steps S210 to S240.
[0088] S210: Acquire a historical data set, where the historical data set includes multiple data items, and a data item is a character sequence composed of multiple characters.
[0089] S220: Generate a character set and a string set corresponding to each sequence position in the character sequence based on the historical data set, wherein the character set is a deduplicated set consisting of characters located at one sequence position, and the string set is a deduplicated set consisting of strings located at at least two sequence positions.
[0090] S230: Filtering a target character string set for abnormal data detection according to the number of characters in the character set and the number of character strings in the character string set.
[0091] S240: Perform consistency matching verification on the data to be detected and the target character string set, and determine that the data to be detected is abnormal data when the verification fails.
[0092] In the abnormal data detection method provided in the embodiment of the present application, by efficiently utilizing the historical data set, a deduplication set containing all characters and strings is first constructed, and then based on the number of characters and strings in these sets, a key target string set for abnormal detection is intelligently screened. This method not only improves the accuracy of abnormal data detection, but also can quickly identify and determine the abnormalities in the data to be detected through consistency matching verification, thereby providing significant benefits in terms of data quality and security, and ensuring the reliability and effectiveness of data processing.
[0093] The following describes in detail the various method steps of the abnormal data detection method in the embodiment of the present application in combination with specific application scenarios.
[0094] In step S210, a historical data set is obtained, where the historical data set includes a plurality of data items, and a data item is a character sequence composed of a plurality of characters.
[0095] Taking the business serial number of the financial business system as an example, a certain financial business (such as bank deposit business) will generate a serial number during the business process. Assume that the serial number consists of a date and 4 random letters, and the time is from January 1, 1900 to March 31, 2024. Among them, each serial number is a data item, which is a character sequence consisting of 8 characters.
[0096] Examples of serial number data are as follows: 19000101ckot, 19000102gnwo, 19000103wamf, 19000104xydn, 19000105crky, 19000106wekm, 19000107hcok, 19000108szgr, 19000109exif, 19000110skua, …, 20240322itah, 20240323vtem, 20240324tysj, 20240325hmgp, 20240326zrac, 20240327iyca, 20240328jsxt, 20240329uzvb, 20240330ersk, 20240331hmbs
[0097] In step S220, based on the historical data set, a character set and a string set corresponding to each sequence position in the character sequence are generated, wherein the character set is a deduplicated set consisting of characters located at one sequence position, and the string set is a deduplicated set consisting of strings located at at least two sequence positions.
[0098] Figure 3 A flowchart of generating a character set and a string set in one embodiment of the present application is shown. Figure 3 As shown, generating a character set and a string set corresponding to each sequence position in the character sequence according to the historical data set in step S220 may further include the following steps S221 to S223.
[0099] S221: Sequence each data item in the historical data set according to single characters and combine them to obtain a character matrix, where a character row in the character matrix corresponds to a data item, and a character column in the character matrix corresponds to a sequence position.
[0100] The characters in each data item are aligned according to the sequence position, split and combined to form a character matrix, in which each matrix point is a character.
[0101] Figure 4 FIG. 2 shows a character matrix in an application scenario of an embodiment of the present application. Figure 4 As shown, each data item is regarded as a character row, and it is split into multiple character columns according to the sequence position to form a character matrix. For the convenience of explanation, AL is used to mark each character column from left to right.
[0102] S222: Remove duplicate characters from all characters in a character column to form a character set corresponding to a sequence position.
[0103] For each character column in the character matrix, a corresponding character set is generated after removing duplicate characters.
[0104] Figure 5 The character set corresponding to each character column in an application scenario of the embodiment of the present application is shown. Figure 5 As shown, the character set corresponding to column A is {1, 2}, and the character set corresponding to column B is {0, 9}.
[0105] S223: According to the arrangement order of the sequence positions, the characters on the two character columns are concatenated to form character strings corresponding to each character row, and duplicate items are removed from the character strings to form a character string set corresponding to the two sequence positions.
[0106] All character strings are concatenated in pairs, and after removing duplicates, a string set consisting of the concatenated strings is generated.
[0107] Figure 6 The following shows a character sequence combination used to concatenate character strings in an application scenario of an embodiment of the present application. Figure 6 As shown, every two character columns are concatenated to form a string, for example, column A and column B are a combination of one concatenated string, and column B and column C are a combination of another concatenated string.
[0108] In one embodiment of the present application, the character string includes an adjacent character string and a non-adjacent character string. The adjacent character string is composed of two characters whose sequence positions are adjacent, and the non-adjacent character string is composed of two characters whose sequence positions are not adjacent.
[0109] Continue to refer Figure 6 As shown in the figure, columns A and B correspond to two adjacent sequence positions, and the corresponding concatenated strings are adjacent strings. Columns A and C correspond to two non-adjacent sequence positions (with column B in between), and the corresponding concatenated strings are non-adjacent strings. Based on non-adjacent strings, the association between non-adjacent sequence positions can be mined to further improve the reliability of abnormal data detection.
[0110] In step S230 , a target character string set for abnormal data detection is screened according to the number of characters in the character set and the number of character strings in the character string set.
[0111] Based on the character column combination in the previous step, remove the duplicates of the string and record the size of each string set, that is, get the number of strings in the corresponding string set. For example, the string set corresponding to columns AB is {19, 20}, and the number of strings in this string set is 2. Correspondingly, the character set corresponding to column A is {1, 9}, and its number of characters is 2; the character set corresponding to column B is {2, 0}, and its number of characters is also 2.
[0112] In one embodiment of the present application, screening a target string set for abnormal data detection based on the number of characters in a character set and the number of strings in a string set may further include: determining the number of character combinations corresponding to two sequence positions based on the number of characters in the character set, the number of character combinations being the maximum number of character combinations generated by free combinations of characters at the two sequence positions; and screening the target string set for abnormal data detection based on a numerical relationship between the number of character combinations and the number of strings.
[0113] For example, the character set corresponding to column A is {1, 9}, and the number of characters is 2; the character set corresponding to column B is {2, 0}, and the number of characters is also 2. According to the number of characters in columns AB, it can be determined that the number of corresponding character combinations is 2*2=4.
[0114] For another example, the character set corresponding to column K is 26 lowercase letters, and the character set corresponding to column L is also 26 lowercase letters. According to the number of characters in columns K and L, the number of corresponding character combinations can be determined to be 26*26=676.
[0115] In one embodiment of the present application, if the number of character combinations is greater than the number of character strings, the character string set corresponding to the number of character strings is determined as a target character string set for abnormal data detection.
[0116] For example, the number of character combinations corresponding to columns AB is 4, and the number of character strings corresponding to the two is 2, so it can be determined that the character string set corresponding to columns AB is the target character string set for abnormal data detection.
[0117] For another example, the number of character combinations in the two columns KL is 676, and the number of corresponding character strings is also 676. Therefore, it can be determined that the two columns KL are not related, that is, the character string set corresponding to the two columns KL should not be selected as the target character string set for abnormal data detection.
[0118] In one embodiment of the present application, when the number of character combinations is less than a preset first quantity threshold, if the number of character combinations is greater than the number of character strings, the string set corresponding to the number of character strings is determined as a target string set for abnormal data detection; when the number of character combinations is greater than the first quantity threshold, a correlation check is performed on two sequence positions corresponding to the number of character combinations, and when the check is successful, the string set corresponding to the number of character strings is determined as a target string set for abnormal data detection.
[0119] For example, assuming that both character sequences are random characters in the ASCII code, the number of characters in the character set corresponding to each character sequence is 128, and the number of character combinations formed by the combination of the two character sequences is 128*128=16384. However, in actual data, all 128*128 combinations may not be covered, which may result in erroneously combining two unrelated character sequences into a target character string set.
[0120] To address this problem, the embodiment of the present application determines whether to perform a correlation check on two sequence positions according to a preset first quantity threshold. For example, the first quantity threshold is 8000. When the number of character combinations corresponding to the two sequence positions is greater than 8000, a correlation check can be performed on the two sequence positions corresponding to the number of character combinations, and when the check succeeds, the character string set corresponding to the number of character strings is determined as the target character string set for abnormal data detection.
[0121] In one embodiment of the present application, a method for checking the association between two sequence positions corresponding to the number of character combinations may include: obtaining a ratio threshold corresponding to the two sequence positions, the ratio threshold being less than 1; if the product of the number of character combinations and the ratio threshold is greater than the number of character strings, determining that the two sequence positions corresponding to the number of character combinations are associated.
[0122] For example, the number of character combinations corresponding to the two columns KL is 16384, which is greater than the first quantity threshold of 8000. At this time, the ratio threshold corresponding to the two sequence positions can be set to 0.9. If 16384*0.9 is greater than the corresponding number of strings, it can be determined that the two sequence positions corresponding to the two columns KL are associated; if 16384*0.9 is less than or equal to the corresponding number of strings, it can be determined that the two sequence positions corresponding to the two columns KL are not associated.
[0123] In one embodiment of the present application, a method for checking the correlation between two sequence positions corresponding to the number of character combinations may include: counting the actual occurrence frequency and theoretical occurrence frequency of any pair of character combinations at the two sequence positions corresponding to the number of character combinations; determining a reliability coefficient for the correlation between the two sequence positions based on the actual occurrence frequency and the theoretical occurrence frequency; if the reliability coefficient is greater than a preset reliability threshold, determining that the two sequence positions corresponding to the number of character combinations are correlated.
[0124] Figure 7 FIG. 2 shows a frequency matrix used to count the frequency of occurrence in an application scenario of an embodiment of the present application. Figure 7 As shown, taking the two columns KL as an example, the values of column K are expanded vertically and the values of column L are expanded horizontally. Combined with the source data set, the frequency of occurrence of each pair of combinations is counted.
[0125] First, assume that columns K and L are two independent character columns that have no correlation, that is, the values of columns K and L do not affect each other. According to the probability multiplication formula of independent events: P(A∩B)=P(A)*P(B), calculate the probability of each KL column combination.
[0126] For the convenience of explanation, select Figure 7 The frequencies of the first three rows and first three columns are calculated. Figure 8 The frequency statistics results of the embodiment of the present application in an application scenario are shown. Figure 8 As shown, the total number of samples obtained by the frequency statistics of the first three rows and the first three columns is 591. Similarly, the frequency statistics of the complete 26-row and 26-column frequency matrix can be performed in the same way.
[0127] according to Figure 8 The frequency statistics shown can be used to calculate the probability of occurrence of each character in turn. Specifically, it includes:
[0128]
[0129]
[0130]
[0131]
[0132]
[0133]
[0134] On this basis, the co-occurrence probability of two characters can be calculated. Specifically:
[0135] P(Ka and La)=P(Ka)*P(La)=0.350254*0.326565=0.114381;
[0136] P(Ka and Lb)=P(Ka)*P(Lb)=0.350254*0.326565=0.1209;
[0137] P(Ka and Lc)=P(Ka)*P(Lc)=0.350254*0.326565=0.114973;
[0138] P(Kb and La)=P(Ka)*P(La)=0.350254*0.326565=0.098909;
[0139] P(Kb and Lb)=P(Ka)*P(Lb)=0.350254*0.326565=0.104546;
[0140] P(Kb and Lc)=P(Ka)*P(Lc)=0.350254*0.326565=0.099421;
[0141] P(Kc and La)=P(Ka)*P(La)=0.350254*0.326565=0.113276;
[0142] P(Kc and Lb) = P(Ka)*P(Lb) = 0.350254*0.326565 = 0.119732;
[0143] P(Kc and Lc)=P(Ka)*P(Lc)=0.350254*0.326565=0.113862.
[0144] By multiplying the above probabilities by the total number of samples 591, we can get the theoretical frequency of occurrence of each pair of character combinations.
[0145] Fig. 9 The statistical result of the occurrence frequency of the character combination in an application scenario of the embodiment of the present application is shown. In each cell, the value outside the brackets is the actual occurrence frequency of the character combination, and the value inside the brackets is the theoretical occurrence frequency of the character combination.
[0146] According to the actual occurrence frequency and theoretical occurrence frequency of the character combination, the reliability coefficient of the correlation between the corresponding two sequence positions can be calculated.
[0147] In one embodiment of the present application, the reliability coefficient is calculated according to the following formula:
[0148]
[0149] Among them, x 2 is the reliability coefficient, y1 is the actual occurrence frequency of the character combination, and y2 is the theoretical occurrence frequency of the character combination.
[0150] by Fig. 9 Taking the frequency statistics shown in the figure as an example, the reliability coefficient is calculated as follows:
[0151]
[0152] When the reliability coefficient is greater than a preset reliability threshold, it can be determined that the two sequence positions corresponding to the number of character combinations are associated.
[0153] In one embodiment of the present application, according to the preset significance level and degree of freedom, a preset reliability threshold data table can be queried to obtain the corresponding reliability threshold. The reliability threshold data table can be, for example, a chi-square value data table. The chi-square value is a statistic in a non-parametric test, mainly used in non-parametric statistical analysis. It is a main test indicator in the chi-square test. The chi-square test is a widely used hypothesis test method for counting data. It belongs to the category of non-parametric tests. It mainly compares two or more sample rates (constituent ratios) and the correlation analysis of two categorical variables. The fundamental idea is to compare the degree of agreement or goodness of fit between the theoretical frequency and the actual frequency.
[0154] For example, for the significance level α = 0.05 and the degree of freedom n = 4, the corresponding reliability threshold can be obtained as
[0155] Similarly, for the complete KL column, the corresponding reliability coefficient can be calculated as 609.3666940883254. For the significance level α = 0.05 and the degree of freedom n = 4, the corresponding reliability threshold can be obtained as
[0156] Since the reliability coefficient is less than the reliability threshold, it can be determined that the two columns KL are two independent character columns that are not related.
[0157] In one embodiment of the present application, the method of performing correlation checking on two sequence positions corresponding to the number of character combinations may further include:
[0158] When the number of character combinations is less than a preset second number threshold, a ratio threshold corresponding to the two sequence positions is obtained, and the ratio threshold is less than 1; if the product of the number of character combinations and the ratio threshold is greater than the number of character strings, it is determined that the two sequence positions corresponding to the number of character combinations are associated;
[0159] When the number of character combinations is greater than a second quantity threshold, the actual occurrence frequency and theoretical occurrence frequency of any pair of character combinations are counted at the two sequence positions corresponding to the number of character combinations; the reliability coefficient of the correlation between the two sequence positions is determined based on the actual occurrence frequency and the theoretical occurrence frequency; if the reliability coefficient is greater than a preset reliability threshold, it is determined that the two sequence positions corresponding to the number of character combinations are correlated.
[0160] For example, the second quantity threshold is 12000. When the number of character combinations is less than 12000, the relevance check can be performed based on the ratio threshold; when the number of character combinations is greater than 12000, the relevance check can be performed based on the reliability coefficient.
[0161] The embodiment of the present application adopts a progressive quantity threshold, which can improve the screening efficiency of the target character string set and reduce the calculation cost.
[0162] by Figure 6 The string set shown is used as an example. The target string set obtained by screening includes [AB, AC, BC, EF, GH]. Among them, AB is the first two digits of the year, AC is the 1st and 3rd digits of the year, BC is the 2nd and 4th digits of the year, EF is the month, and GH is the date 1-31, which has a certain regularity.
[0163] In one embodiment of the present application, after the target character string sets for abnormal data detection are screened, when any two target character string sets correspond to a same sequence position, the two target character string sets are merged.
[0164] Since the ultimate goal is to segment the regular data and save the deduplicated set after segmentation as the basis for identifying abnormal data, we need to continue looking for the combined constraint set of the maximum length. However, at this time, we do not use the method of traversing the data set again, but use the probabilistic derivation method to merge them.
[0165] For example, for three events A, B and C, if the probabilities of A and B are known, and the probabilities of B and C are known, the probabilities of A and C can be deduced from this information. This is the transitivity of probability.
[0166] Specifically, if the probability of event A and event B occurring at the same time (denoted as P(A∩B)), as well as the probability of event B and event C occurring at the same time (denoted as P(B∩C)) are known, the probability of event A and event C occurring at the same time (denoted as P(A∩C)) can be calculated using this information.
[0167] The key here is to understand the transitivity of probability, that is, if the joint probability of two events is known, and the joint probability of one of the events with a third event, the joint probability of the other two events can be inferred.
[0168] For example, P(A∩B) = known probability, i.e., the probability of events A and B occurring simultaneously. P(B∩C) = known probability, i.e., the probability of events B and C occurring simultaneously.
[0169] To find P(A∩C), the probability of events A and C occurring simultaneously, use the following steps:
[0170] First, calculate the conditional probability of event B occurring when event A occurs, that is, P(B|A). By definition, P(B|A) = P(A∩B) / P(A).
[0171] Then, use P(B|A) to calculate the conditional probability of event C occurring when event A occurs, which is P(C|A). This can be calculated using the following formula: P(C|A) = P(B∩C) / P(B|A).
[0172] Finally, P(A) and P(C|A) can be used to calculate P(A∩C). According to the total probability formula, P(A∩C) = P(A)*P(C|A).
[0173] This process shows how to derive new joint probabilities from known joint probabilities and conditional probabilities, that is, the transitivity of probability.
[0174] According to the transitivity rule, two-column combinations can be further merged to obtain three-column combinations, four-column combinations, five-column combinations, and so on, until no longer combination constraints can be derived.
[0175] For example, among the combined constraints that have been mined, AB, AC, and BC are mutually constrained. These rules that form mutual constraints are merged into a new combined constraint rule ABC, which corresponds to a target string set consisting of strings of length 3.
[0176] Based on the pairwise combination constraints [AB, AC, BC, EF, GH], the updated combination constraints [ABC, EF, GH] can be merged, where each combination constraint corresponds to a target string set.
[0177] Based on the best segmentation combination calculated based on the current actual data, the character sequence ABCDEFGHIJKL can be segmented according to the longest combination constraint set [ABC, EF, GH] to obtain the corresponding character sequence segmentation result [ABC, D, EF, GH, I, J, K, L], and then the corresponding multiple target character string sets can be obtained.
[0178] Fig.10The embodiment of the present application shows multiple target character string sets formed by segmenting a character sequence in an application scenario. So far, hundreds of millions of serial numbers have been compressed into a simple combination rule for storage and application.
[0179] In step S240, the data to be detected is checked for consistency with the target character string set, and when the check fails, the data to be detected is determined to be abnormal data.
[0180] Fig.11 A schematic diagram showing a consistency match check between the data to be detected and the target string set in an application scenario according to an embodiment of the present application is shown.
[0181] like Fig.11 As shown, there is a piece of data to be detected, 20200132abcd, and the character sequence is segmented in the manner of [ABC, D, EF, GH, I, J, K, L], and then checked one by one with the target string set. Since the GH column combination 32 does not appear in the corresponding target string set, it can be determined that the data to be detected is abnormal data.
[0182] Based on the introduction of the above application scenarios, it can be known that the embodiments of the present application are massive data for big data platforms, and the data involved is at the trillion level. If the traditional Apriori algorithm is used, if the length of the combined rule set to be mined is 10, then the number of scans of the data is 10 trillion times, while the embodiments of the present application only need to scan 1 trillion times. The embodiments of the present application support the free combination of column values while maintaining column alignment, which is not taken into account by the traditional Apriori algorithm. When the amount of data is large enough, the embodiments of the present application will form the rules of random combination trends and mine them using probability distribution, thereby improving the detection efficiency and detection success rate of abnormal data.
[0183] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0184] The following introduces an apparatus embodiment of the present application, which can be used to execute the abnormal data detection method in the above-mentioned embodiment of the present application. Fig.12 The structure block diagram of the abnormal data detection device provided in the embodiment of the present application is schematically shown. Fig.12 As shown, the abnormal data detection device 1200 includes:
[0185] The acquisition module 1210 is configured to acquire a historical data set, wherein the historical data set includes a plurality of data items, and the data items are character sequences composed of a plurality of characters;
[0186] A generating module 1220 is configured to generate, based on the historical data set, a character set and a string set corresponding to each sequence position in the character sequence, wherein the character set is a duplicate-free set consisting of characters located at one sequence position, and the string set is a duplicate-free set consisting of strings located at at least two sequence positions;
[0187] A screening module 1230 is configured to screen a target character string set for abnormal data detection according to the number of characters in the character set and the number of character strings in the character string set;
[0188] The verification module 1240 is configured to perform a consistency matching verification on the data to be detected and the target character string set, and determine that the data to be detected is abnormal data when the verification fails.
[0189] In some embodiments of the present application, based on the above technical solution, the generation module includes:
[0190] A splitting module is configured to split each data item in the historical data set into a sequence according to a single character and combine them to obtain a character matrix, wherein a character row in the character matrix corresponds to a data item, and a character column in the character matrix corresponds to a sequence position;
[0191] A combining module, configured to remove duplicates from all characters in a character column to form a character set corresponding to a position in the sequence;
[0192] The concatenation module is configured to concatenate characters on two character columns according to the arrangement order of the sequence positions to form character strings corresponding to each character row, and to remove duplicates from the character strings to form a character string set corresponding to the two sequence positions.
[0193] In some embodiments of the present application, based on the above technical solution, the character string includes an adjacent character string and a non-adjacent character string, the adjacent character string is composed of the concatenation of two characters whose sequence positions are adjacent, and the non-adjacent character string is composed of the concatenation of two characters whose sequence positions are not adjacent.
[0194] In some embodiments of the present application, based on the above technical solution, the screening module includes:
[0195] a quantity determination module configured to determine the number of character combinations corresponding to two sequence positions according to the number of characters in the character set, the number of character combinations being the maximum number of character combinations generated by freely combining the characters at the two sequence positions;
[0196] The numerical comparison module is configured to screen a target character string set for abnormal data detection according to a numerical relationship between the number of character combinations and the number of character strings.
[0197] In some embodiments of the present application, based on the above technical solution, the numerical comparison module includes:
[0198] A first comparison module is configured to, when the number of character combinations is less than a preset first number threshold, determine a character string set corresponding to the number of character strings as a target character string set for abnormal data detection if the number of character combinations is greater than the number of character strings;
[0199] The second comparison module is configured to perform a correlation check on two sequence positions corresponding to the number of character combinations when the number of character combinations is greater than the first number threshold, and determine the string set corresponding to the number of character strings as a target string set for abnormal data detection when the check is successful.
[0200] In some embodiments of the present application, based on the above technical solution, the second comparison module is also configured to: obtain a ratio threshold corresponding to the two sequence positions, and the ratio threshold is less than 1; if the product of the number of character combinations and the ratio threshold is greater than the number of character strings, then determine that the two sequence positions corresponding to the number of character combinations are associated.
[0201] In some embodiments of the present application, based on the above technical solution, the second comparison module is further configured to: count the actual occurrence frequency and theoretical occurrence frequency of any pair of character combinations at the two sequence positions corresponding to the number of character combinations; determine the reliability coefficient of the correlation between the two sequence positions based on the actual occurrence frequency and the theoretical occurrence frequency; if the reliability coefficient is greater than a preset reliability threshold, it is determined that the two sequence positions corresponding to the number of character combinations are correlated.
[0202] In some embodiments of the present application, based on the above technical solution, the reliability coefficient is calculated according to the following formula:
[0203]
[0204] Among them, x 2 is the reliability coefficient, y1 is the actual occurrence frequency of the character combination, and y2 is the theoretical occurrence frequency of the character combination.
[0205] In some embodiments of the present application, based on the above technical solution, the second comparison module is further configured to: when the number of character combinations is less than a preset second number threshold, obtain a ratio threshold corresponding to the two sequence positions, and the ratio threshold is less than 1; if the product of the number of character combinations and the ratio threshold is greater than the number of character strings, determine that the two sequence positions corresponding to the number of character combinations are associated; when the number of character combinations is greater than the second number threshold, count the actual occurrence frequency and theoretical occurrence frequency of any pair of character combinations at the two sequence positions corresponding to the number of character combinations; determine the reliability coefficient of the association between the two sequence positions based on the actual occurrence frequency and the theoretical occurrence frequency; if the reliability coefficient is greater than the preset reliability threshold, determine that the two sequence positions corresponding to the number of character combinations are associated.
[0206] In some embodiments of the present application, based on the above technical solution, the device further includes:
[0207] The merging module is configured to merge any two target character string sets when the two target character string sets correspond to a same sequence position.
[0208] The specific details of the abnormal data detection device provided in each embodiment of the present application have been described in detail in the corresponding method embodiment and will not be repeated here.
[0209] Fig.13 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.
[0210] It should be noted that Fig.13 The computer system 1300 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0211] like Fig.13As shown, the computer system 1300 includes a central processing unit 1301 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1302 (ROM) or the program loaded from the storage part 1308 to the random access memory 1303 (RAM). Various programs and data required for system operation are also stored in the random access memory 1303. The central processing unit 1301, the read-only memory 1302 and the random access memory 1303 are connected to each other through a bus 1304. The input / output interface 1305 (Input / Output interface, i.e., I / O interface) is also connected to the bus 1304.
[0212] The following components are connected to the input / output interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the input / output interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed so that a computer program read therefrom is installed into the storage section 1308 as needed.
[0213] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1309, and / or installed from the removable medium 1311. When the computer program is executed by the central processor 1301, various functions defined in the system of the present application are executed.
[0214] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0215] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0216] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.
[0217] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.
[0218] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.
[0219] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for detecting abnormal data, characterized in that: include: Acquire a historical data set, wherein the historical data set includes a plurality of data items, and the data items are character sequences composed of a plurality of characters; Generate a character set and a string set corresponding to each sequence position in the character sequence according to the historical data set, wherein the character set is a duplicate-free set consisting of characters located at one sequence position, and the string set is a duplicate-free set consisting of strings located at at least two sequence positions; Filtering a target character string set for abnormal data detection according to the number of characters in the character set and the number of character strings in the character string set; The data to be detected is checked for consistency matching with the target character string set, and when the check fails, the data to be detected is determined to be abnormal data.
2. The abnormal data detection method according to claim 1, characterized in that: Generating a character set and a character string set corresponding to each sequence position in the character sequence according to the historical data set, including: Sequentially splitting each data item in the historical data set into individual characters and combining them to obtain a character matrix, wherein a character row in the character matrix corresponds to a data item, and a character column in the character matrix corresponds to a sequence position; Remove duplicate characters from all characters in a character column to form a character set corresponding to a position in the sequence; According to the arrangement order of the sequence positions, the characters on the two character columns are concatenated to form character strings corresponding to each character row, and the character strings are removed of duplicates to form a set of character strings corresponding to the two sequence positions.
3. The abnormal data detection method according to claim 2, characterized in that: The character string includes an adjacent character string and a non-adjacent character string. The adjacent character string is composed of two characters whose sequence positions are adjacent, and the non-adjacent character string is composed of two characters whose sequence positions are not adjacent.
4. The abnormal data detection method according to any one of claims 1 to 3, characterized in that: According to the number of characters in the character set and the number of strings in the string set, a target string set for abnormal data detection is screened, including: Determine the number of character combinations corresponding to the two sequence positions according to the number of characters in the character set, wherein the number of character combinations is the maximum number of character combinations generated by freely combining the characters at the two sequence positions; According to the numerical relationship between the number of character combinations and the number of character strings, a target character string set for abnormal data detection is screened.
5. The abnormal data detection method according to claim 4, characterized in that: According to the numerical relationship between the number of character combinations and the number of character strings, a target character string set for abnormal data detection is screened, including: When the number of character combinations is less than a preset first number threshold, if the number of character combinations is greater than the number of character strings, a character string set corresponding to the number of character strings is determined as a target character string set for abnormal data detection; When the number of character combinations is greater than the first number threshold, a correlation check is performed on two sequence positions corresponding to the number of character combinations, and when the check succeeds, the character string set corresponding to the number of character strings is determined as a target character string set for abnormal data detection.
6. The abnormal data detection method according to claim 5, characterized in that: Performing a correlation check on two sequence positions corresponding to the number of character combinations, including: Obtaining a ratio threshold value corresponding to the two sequence positions, wherein the ratio threshold value is less than 1; If the product of the number of character combinations and the ratio threshold is greater than the number of character strings, it is determined that two sequence positions corresponding to the number of character combinations are associated.
7. The abnormal data detection method according to claim 5, characterized in that: Performing a correlation check on two sequence positions corresponding to the number of character combinations, including: Counting the actual occurrence frequency and theoretical occurrence frequency of any pair of character combinations at two sequence positions corresponding to the number of character combinations; Determining a reliability coefficient of the correlation between the two sequence positions according to the actual occurrence frequency and the theoretical occurrence frequency; If the reliability coefficient is greater than a preset reliability threshold, it is determined that the two sequence positions corresponding to the number of character combinations are associated.
8. The abnormal data detection method according to claim 7, characterized in that: The reliability coefficient is calculated according to the following formula: Among them, x 2 is the reliability coefficient, y1 is the actual occurrence frequency of the character combination, and y2 is the theoretical occurrence frequency of the character combination.
9. The abnormal data detection method according to claim 5, characterized in that: Performing a correlation check on two sequence positions corresponding to the number of character combinations, including: When the number of character combinations is less than a preset second number threshold, a ratio threshold corresponding to the two sequence positions is obtained, and the ratio threshold is less than 1; if the product of the number of character combinations and the ratio threshold is greater than the number of character strings, it is determined that the two sequence positions corresponding to the number of character combinations are associated; When the number of character combinations is greater than the second number threshold, the actual occurrence frequency and theoretical occurrence frequency of any pair of character combinations are counted at the two sequence positions corresponding to the number of character combinations; the reliability coefficient of the correlation between the two sequence positions is determined based on the actual occurrence frequency and the theoretical occurrence frequency; if the reliability coefficient is greater than the preset reliability threshold, it is determined that the two sequence positions corresponding to the number of character combinations are correlated.
10. The abnormal data detection method according to any one of claims 1 to 3, characterized in that: After screening the target string set for abnormal data detection, the method further includes: When any two target character string sets correspond to a same sequence position, the two target character string sets are merged.
Citation Information
Cited By
Method, equipment and product for testing support reliability of database to character set
CN120011244A
