Abnormal data determination method and system
By acquiring and classifying data from multiple data systems, and determining their distribution characteristics and anomalies, the problems of data integration and real-time detection are solved, thereby improving data quality and analysis efficiency.
Patent Information
- Application Number
- CN202510957386.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-11
- Publication Date
- 2025-10-31
AI Technical Summary
Existing technologies struggle to integrate data from multiple data systems and detect anomalous data in real time, impacting data quality and subsequent analysis and decision-making.
Data to be classified is obtained from multiple different data systems, classified according to the attribute information of the data, the data distribution characteristics of the dataset to be processed are determined, and whether the data is abnormal is determined by the benchmark value and anomaly score, and an alert is issued.
It enables unified storage and anomaly detection of data from multiple different systems, improving data quality and preventing abnormal data from affecting subsequent analysis and decision-making.
Smart Images
Figure CN120873676A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data management technology, and in particular to a method and system for identifying abnormal data. Background Technology
[0002] In today's information age, data has become one of the most crucial core assets for enterprises. With the rapid advancement of technology, information systems within enterprises and with external partners are becoming increasingly diversified, encompassing various technical architectures and data formats. However, this diversity poses a significant challenge to information technology management in terms of data sharing and interaction.
[0003] Data management methods in general technologies often struggle to efficiently process real-time data. Updating and storing real-time data involves handling massive data streams and requires real-time detection and alerts for anomalies during data storage. A lack of anomaly detection and alerts can negatively impact the quality of stored data, adversely affecting subsequent data analysis and decision-making. Furthermore, traditional data management methods are often limited to a single data source, severely restricting the ability to integrate data from multiple systems.
[0004] Therefore, how to integrate data from multiple systems and perform real-time anomaly detection on stored data is an urgent problem to be solved. Summary of the Invention
[0005] This disclosure provides a method and system for determining abnormal data, aiming to solve the technical problem that it is impossible to determine abnormal data from multiple data systems in general technology.
[0006] To achieve the above objectives, this application adopts the following technical solution:
[0007] Firstly, a method for identifying anomalous data is provided, comprising: acquiring data to be classified from multiple different data systems; classifying the data to be classified according to the attribute information of the data to be classified, and determining at least one category of dataset to be processed; acquiring the data distribution characteristics of the dataset to be processed for each category of dataset to be processed; the dataset to be processed includes data of the same business type from multiple different data systems; and determining whether the data to be processed is anomalous data based on the data to be processed in the dataset to be processed and the data distribution characteristics of the dataset to be processed.
[0008] Optionally, based on the data to be processed in the dataset to be processed and the data distribution characteristics of the dataset to be processed, it is determined whether the data to be processed is abnormal data, including: determining a baseline value for the data to be processed based on the data distribution characteristics of the data to be processed and the dataset to be processed; the baseline value is used to represent the business accuracy and / or numerical accuracy of the data to be processed; if the baseline value of the data to be processed is not within a preset range, the data to be processed is determined to be abnormal data; if the baseline value of the data to be processed is within a preset range, the data to be processed is determined not to be abnormal data.
[0009] Optionally, after determining that the data to be processed is abnormal data when the baseline value of the data to be processed is not within the preset range, the method further includes: determining the basic score of the dataset to be processed based on the baseline value of the abnormal data; the basic score is positively correlated with the baseline value; determining the abnormal score of the dataset to be processed based on the basic score; and issuing an early warning for the dataset to be processed when the abnormal score of the dataset to be processed is greater than the preset abnormal score.
[0010] Optionally, based on the baseline score, an anomaly score for the dataset to be processed is determined, including: the storage permission level for obtaining the anomaly data; and the anomaly score for the dataset to be processed is determined based on the storage permission level of the anomaly data and the baseline score.
[0011] Optionally, based on the storage permission level of the abnormal data and the base score, the abnormal score of the dataset to be processed is determined, including: if the storage permission level of the abnormal data is less than the preset permission level, the base score is determined as the abnormal score; if the storage permission level of the abnormal data is greater than or equal to the preset permission level, a target adjustment coefficient is determined based on the storage permission level of the abnormal data, and the base score is adjusted based on the target adjustment coefficient to obtain the abnormal score.
[0012] Optionally, a target adjustment coefficient is determined based on the storage permission level of the abnormal data, including: determining the level difference between the storage permission level of the abnormal data and the preset permission level; determining the ratio between the level difference and the preset level difference as the initial adjustment coefficient; obtaining the amount of data duplication in the dataset to be processed; and determining the target adjustment coefficient based on the amount of data duplication and the initial adjustment coefficient.
[0013] Optionally, a target adjustment coefficient is determined based on the data repetition amount and the initial adjustment coefficient, including: determining the initial adjustment coefficient as the target adjustment coefficient when the data repetition amount is less than the preset data repetition amount, or when the difference in quantity between the data repetition amount and the preset data repetition amount is less than the preset quantity difference; and determining a correction coefficient to adjust the initial adjustment coefficient when the data repetition amount is greater than or equal to the preset data repetition amount, and when the difference in quantity between the data repetition amount and the preset data repetition amount is greater than or equal to the preset quantity difference, and adjusting the initial adjustment coefficient according to the correction coefficient to obtain the target adjustment coefficient.
[0014] Optionally, a correction factor is determined to adjust the initial adjustment factor, including: determining the correction factor based on the difference between the data repetition amount and the preset data repetition amount and the total amount of data in the dataset to be processed; the correction factor is negatively correlated with the data repetition amount.
[0015] Optionally, an alert is issued for the dataset to be processed, including: determining the difference between the anomaly score and the preset anomaly score; determining the alert level based on the difference between the anomaly score difference and the preset alert score difference; the alert level is positively correlated with the difference between the anomaly score difference and the preset alert score difference; and issuing an alert for the dataset to be processed based on the alert level.
[0016] Secondly, an abnormal data determination system is provided, which includes an acquisition module and a storage module.
[0017] The acquisition module is used to acquire data to be classified from multiple different data systems.
[0018] The storage module is used to classify the data to be classified based on the attribute information of the data to be classified, and to determine at least one category of the dataset to be processed.
[0019] The storage module is also used to obtain the data distribution characteristics of the dataset to be processed for each category.
[0020] The storage module is also used to determine whether the data to be processed is abnormal based on the data to be processed in the dataset and the data distribution characteristics of the dataset.
[0021] Thirdly, an abnormal data determination device is provided, comprising: a memory and a processor; the memory is used to store computer execution instructions, and the processor is connected to the memory via a bus; when the abnormal data determination device is running, the processor executes the computer execution instructions stored in the memory, so that the abnormal data determination device performs the abnormal data determination method of the first aspect.
[0022] The device for determining abnormal data can be an electronic device or a component of an electronic device, such as a chip system within the electronic device. This chip system supports the electronic device in implementing the functions involved in the first aspect and any possible implementation thereof, such as acquiring and determining the data and / or information involved in the aforementioned method for determining abnormal data. The chip system includes a chip, but may also include other discrete devices or circuit structures.
[0023] Fourthly, a computer-readable storage medium is provided, comprising computer-executable instructions that, when executed on a computer, cause the computer to perform the abnormal data determination method described in the first aspect.
[0024] Fifthly, a computer program product is also provided, comprising a computer program or instructions that, when executed on an abnormal data determination device, cause the abnormal data determination device to perform the abnormal data determination method as described in the first aspect above.
[0025] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the abnormal data determination device, or it may be packaged separately from the processor of the abnormal data determination device; this application does not limit this.
[0026] The descriptions of the second, third, fourth, and fifth aspects of this application can be referenced to the detailed description of the first aspect.
[0027] In the embodiments of this application, the name of the aforementioned abnormal data determination system does not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. For example, a storage module may also be called a storage unit, memory, etc. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalents.
[0028] The technical solution provided in this application brings at least the following beneficial effects:
[0029] Based on any of the above aspects, embodiments of this application provide a method for determining abnormal data, comprising: first, acquiring data to be classified from multiple different data systems; second, classifying the data to be classified according to attribute information to determine at least one category of datasets to be processed; then, acquiring the data distribution characteristics of the datasets to be processed for each category. The datasets to be processed include data of the same business type from multiple different data systems. Subsequently, determining whether the data to be processed is abnormal data based on the data to be processed in the datasets and the data distribution characteristics of the datasets to be processed.
[0030] As described above, this application acquires and classifies data from multiple different systems. It then performs anomaly detection on the classified dataset, identifying anomalous data based on the dataset and the data to be processed within it. This achieves unified storage and anomaly detection for data from multiple systems. Anomaly detection across multiple systems improves data quality, preventing the impact of anomalous data during subsequent data analysis and strategy generation.
[0031] The beneficial effects of the first, second, third, fourth, and fifth aspects of this application can all be referred to in the analysis of the above-mentioned beneficial effects, and will not be repeated here. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of the structure of an abnormal data determination system provided in an embodiment of this application;
[0033] Figure 2 A schematic diagram of the structure of another abnormal data determination system provided in the embodiments of this application;
[0034] Figure 3 A schematic diagram of the hardware structure of an abnormal data determination device provided in an embodiment of this application;
[0035] Figure 4 A flowchart illustrating an abnormal data determination method provided in an embodiment of this application;
[0036] Figure 5 A flowchart illustrating another method for determining abnormal data provided in this application embodiment;
[0037] Figure 6 A flowchart illustrating another method for determining abnormal data provided in this application embodiment;
[0038] Figure 7 A flowchart illustrating another method for determining abnormal data provided in this application embodiment;
[0039] Figure 8 A flowchart illustrating another method for determining abnormal data provided in this application embodiment;
[0040] Figure 9 A flowchart illustrating another method for determining abnormal data provided in this application embodiment;
[0041] Figure 10 A flowchart illustrating another method for determining abnormal data provided in this application embodiment;
[0042] Figure 11 A flowchart illustrating another method for determining abnormal data provided in this application embodiment;
[0043] Figure 12 This is a schematic diagram of an abnormal data determination device provided in an embodiment of this application. Detailed Implementation
[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0045] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0046] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.
[0047] Before providing a detailed introduction to the method for determining abnormal data provided in this application, let's briefly introduce the application scenarios and implementation environment involved in this application.
[0048] First, a brief introduction to the application scenarios involved in this application will be given.
[0049] As described in the background section, general data management methods are insufficient for integrating data from multiple systems. Furthermore, data storage requires anomaly detection to ensure data quality.
[0050] To address the aforementioned issues, this application provides a method for determining abnormal data, comprising: first, acquiring data to be classified from multiple different data systems; second, classifying the data to be classified based on its attribute information to determine at least one category of datasets to be processed; then, acquiring the data distribution characteristics of each category of datasets to be processed. The datasets to be processed include data of the same business type from multiple different data systems. Subsequently, based on the data to be processed in the datasets and the data distribution characteristics of the datasets to be processed, determining whether the data to be processed is abnormal data.
[0051] As described above, this application acquires data to be classified from multiple different systems and categorizes and stores this data. Then, it performs anomaly detection on the dataset to be processed after classification and storage, identifying anomalous data based on the dataset and the data to be processed within it. This achieves unified storage and anomaly detection for data from multiple different systems. By performing anomaly detection on the data stored from multiple different systems, the quality of the stored data can be improved, thus avoiding the impact of anomalous data when performing data analysis and strategy generation based on the stored data from multiple different systems.
[0052] The implementation environment for the above-mentioned abnormal data determination method can be the abnormal data determination system provided in the embodiments of this application.
[0053] Figure 1 A schematic diagram of the structure of an abnormal data determination system provided in an embodiment of this application is shown. Figure 1 As shown, the abnormal data determination system includes: an abnormal data determination device 101 and a data storage device 102.
[0054] The abnormal data determination device 101 includes: an acquisition module 103 and a storage module 104.
[0055] Specifically, there is a communication connection between the acquisition module 103 and the storage module 104.
[0056] In practical applications, the abnormal data determination device 101 can be connected to any number of data storage devices 102. For ease of understanding, Figure 1 The following is an example of how an abnormal data identification device 101 is connected to a data storage device 102.
[0057] In this embodiment, the data storage device 102 stores the dataset to be processed determined by the abnormal data determination device 101 after classifying the acquired data to be classified. It also provides the stored dataset to the abnormal data determination device 101 so that the device can determine the abnormal data within the dataset.
[0058] In this embodiment, the acquisition module 103 is used to acquire data to be classified from multiple different data systems. The data to be classified can be real-time data from each data system. The storage module 104 is used to classify the data to be classified acquired by the acquisition module 103 to determine at least one category of the dataset to be processed. The storage module 104 is also used to acquire the data distribution characteristics of the dataset to be processed for each category. The storage module 104 is further used to determine whether the data to be processed is abnormal data based on the data to be processed in the dataset and the data distribution characteristics of the dataset.
[0059] As can be seen, the embodiments of this application can obtain data to be classified from multiple different data systems through the acquisition module 103, thereby realizing unified management of data from multiple different systems.
[0060] Secondly, while the storage module 104 categorizes and stores the data to be classified acquired by the acquisition module 103, it also detects abnormal data in the stored data and determines the abnormal data score and abnormal warning level. This allows for timely detection and handling of data anomalies in the stored dataset, preventing abnormal data from negatively impacting the overall quality of the stored data.
[0061] In some embodiments, combined with Figure 1 ,like Figure 2 As shown, Figure 2 This is a schematic diagram of another abnormal data determination system provided in an embodiment of this application. The system further includes an analysis module 105 and a display module 106.
[0062] The storage module 104 is connected to the analysis module 105, and the analysis module 105 is connected to the display module 106.
[0063] Optionally, the analysis module 105 generates a data report based on the dataset to be processed stored in the storage module 104. The display module 106 is used to display the data report generated by the analysis module 105 and the abnormal data in the dataset to be processed determined by the storage module 104.
[0064] As can be seen, this embodiment of the application generates data reports through the analysis module 105 based on the data stored in the storage module 104, providing users with comprehensive data analysis functions. It not only helps users intuitively understand data trends and business conditions, but also supports in-depth data mining and decision analysis. Users can more efficiently identify business opportunities and potential problems and optimize business decisions through the generated reports.
[0065] Secondly, the display module 106 not only visualizes the data reports generated by the analysis module 105, but also intuitively displays abnormal data when the storage module 104 issues an alert. This function enhances the user experience of the system, making data display and anomaly handling more intuitive and easier to operate. Users can obtain anomaly alerts and data displays in real time, improving the response speed and decision-making efficiency to data anomalies.
[0066] Optionally, the physical device of the abnormal data determination device 101 can be a server, a terminal, or other types of electronic devices, and this application embodiment does not limit this.
[0067] Optionally, the aforementioned terminal may be a device that provides voice and / or data connectivity to a user, a handheld device with wireless connectivity, or other processing device connected to a wireless modem. The wireless terminal may communicate with one or more core networks via a radio access network (RAN). The wireless terminal may be a mobile terminal, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal, or a portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile device that exchanges voice and / or data with the radio access network, such as a mobile phone, tablet computer, laptop computer, netbook, or personal digital assistant (PDA).
[0068] Optionally, the server mentioned above can be one of the servers in a server cluster (composed of multiple servers), a chip in the server, a system-on-a-chip in the server, or a virtual machine (VM) deployed on a physical machine. This application embodiment does not limit this.
[0069] Optionally, the abnormal data determination device 101 and the data storage device 102 can be two independently configured devices, or they can be integrated into the same device. When the abnormal data determination device 101 and the data storage device 102 are integrated into the same device, the data storage device 102 can be the data storage module (e.g., a database) of the abnormal data determination device 101.
[0070] It is easy to understand that when the abnormal data determination device 101 and the data storage device 102 are integrated into the same device, the communication method between the abnormal data determination device 101 and the data storage device 102 is the same as the communication method between internal modules of the device. In this case, the communication process between the two is the same as when the abnormal data determination device 101 and the data storage device 102 are independent of each other.
[0071] For ease of understanding, this application will be described using the example of anomaly data determination device 101 and data storage device 102 being independent of each other.
[0072] The abnormal data determination device 101 in the abnormal data determination system includes, for example: Figure 3 The included components. The following are examples... Figure 3 Taking the abnormal data determination device shown as an example, the hardware structure of the abnormal data determination device 101 is introduced.
[0073] Figure 3 This is a schematic diagram of the hardware structure of an abnormal data determination device provided in an embodiment of this application. Figure 3 As shown, the abnormal data determination device includes a processor 31, a memory 32, a communication interface 33, and a bus 34. The processor 31, the memory 32, and the communication interface 33 can be connected via the bus 34.
[0074] Processor 31 is the control center of the abnormal data determination device. It can be a single processor or a collective term for multiple processing elements. For example, processor 31 can be a general-purpose central processing unit (CPU) or other general-purpose processors. Among them, the general-purpose processor can be a microprocessor or any conventional processor.
[0075] As one embodiment, processor 31 may include one or more CPUs, for example Figure 3 CPU0 and CPU1 are shown in the diagram.
[0076] The memory 32 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.
[0077] In one possible implementation, the memory 32 can exist independently of the processor 31. The memory 32 can be connected to the processor 31 via a bus 34 and is used to store instructions or program code. When the processor 31 calls and executes the instructions or program code stored in the memory 32, it can implement the abnormal data determination method provided in the following embodiments of this application.
[0078] In this embodiment, the software programs stored in memory 32 differ for the abnormal data determination device, resulting in different functions implemented by the device. The functions performed by each device will be described in conjunction with the following flowchart.
[0079] In another possible implementation, the memory 32 can also be integrated with the processor 31.
[0080] Communication interface 33 is used for the abnormal data determination device to connect with other devices via a communication network, which may be Ethernet, wireless access network, wireless local area network (WLAN), etc. Communication interface 33 may include a receiving unit for receiving data and a sending unit for sending data.
[0081] Bus 34 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0082] It should be pointed out that, Figure 3 The structure shown does not constitute a limitation on the abnormal data determination device, except Figure 3 In addition to the components shown, the abnormal data determination device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.
[0083] The abnormal data determination method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0084] The abnormal data determination method provided in this application embodiment is applied to... Figure 1 The abnormal data determination device 101 in the abnormal data determination system shown is as follows: Figure 4 As shown, the abnormal data determination method provided in this application includes:
[0085] S401. Obtain the data to be classified from multiple different data systems.
[0086] Optionally, by acquiring data to be classified from multiple different data systems, the data to be classified from these multiple different data systems can be managed in a unified manner, thereby achieving cross-system data integration.
[0087] For example, in the scenario of acquiring process data, the acquisition module 103 can connect to the data storage units of multiple different data systems (which may include field hardware data acquisition systems and process business management systems with the same business) through a standardized data interface, thereby acquiring the data to be classified (i.e., process data stored in multiple different data systems) from multiple different data systems in real time.
[0088] As can be seen, by acquiring stored data from multiple different data systems through module 103, a foundation is laid for subsequent data storage and anomaly analysis of data from these systems. This effectively solves the challenge of data integration across multiple systems, allowing data from different sources to be uniformly incorporated into the management platform, avoiding the inconsistencies or omissions in data information from multiple data systems caused by processing a single data source in general technologies. Furthermore, by acquiring data from multiple different data systems in real time through module 103, the timeliness of the data is ensured, facilitating timely understanding and response to the latest developments in each system.
[0089] S402. Classify the data to be classified according to the attribute information of the data to be classified, and determine at least one category of the dataset to be processed.
[0090] Specifically, the data to be classified is classified according to its attribute information, and data of the same category are combined into a single dataset for processing.
[0091] Optionally, the attribute information of the data to be classified may include at least one of the following: data source, data type, and data acquisition time.
[0092] For example, when classifying process data obtained from multiple different data systems that store process data, the data can be classified according to data attributes such as the data source (i.e., which data system the process data was obtained from), data type (which may include the business type of the data and the value type of the data), and data acquisition time. Process data of the same category can be divided into the same dataset to be processed, and all datasets to be processed can be stored subsequently, thereby realizing the classification and storage of data from multiple different data systems.
[0093] As can be seen, by uniformly classifying and processing data from multiple different data systems, the embodiments of this application can group data with the same business attributes from multiple different data systems into a dataset to be processed, thereby achieving data integration from multiple different data systems.
[0094] S403. For each category of the dataset to be processed, obtain the data distribution characteristics of the dataset to be processed.
[0095] The dataset to be processed includes data of the same business type from multiple different data systems.
[0096] Specifically, it is necessary to determine whether there is abnormal data in the dataset to be processed based on the data distribution characteristics of the dataset to be processed. Therefore, it is necessary to obtain the data distribution characteristics of the dataset to be processed.
[0097] Optionally, the data distribution characteristics of the dataset to be processed may include the mean and standard deviation of the dataset.
[0098] For example, statistical formulas can be used to calculate feature values such as the mean and standard deviation of the data in the dataset to be processed, which can reflect the distribution characteristics of the data in the dataset. These feature values can be used to intuitively analyze the distribution of the data in the dataset to be processed, providing a basis for subsequent identification of abnormal data.
[0099] S404. Based on the data to be processed in the dataset and the data distribution characteristics of the dataset to be processed, determine whether the data to be processed is abnormal data.
[0100] Specifically, the data distribution characteristics of the dataset to be processed can reflect the data distribution in the dataset. Based on the distribution of the data to be processed in the dataset, it can be determined whether there are abnormal fluctuations in the data to be processed (i.e. the data to be processed exceeds the normal data distribution range), thereby determining whether the data to be processed is abnormal data.
[0101] Optionally, the data to be processed in the dataset to be processed can be anomalous data that has been initially filtered out from the dataset through anomalous data processing.
[0102] In some embodiments, combined with Figure 4 ,like Figure 5 As shown, in S404 above, determining whether the data to be processed is abnormal data based on the data to be processed in the dataset and the data distribution characteristics of the dataset to be processed specifically includes:
[0103] S501. Determine the baseline value of the data to be processed based on the data distribution characteristics of the data to be processed and the dataset to be processed.
[0104] The benchmark value is used to represent the business accuracy and / or numerical accuracy of the data to be processed.
[0105] Optionally, a baseline value for the data to be processed can be determined based on the ratio of the difference between the mean of the data to be processed and the mean of the dataset to be processed to the standard deviation of the dataset to be processed.
[0106] As can be seen, the baseline value of the data to be processed provides a clear indication of the degree of deviation of the data from the dataset, thus allowing for the assessment of the accuracy of the data. Therefore, the baseline value of the data to be processed can be used to determine whether the data is anomaly.
[0107] S502. If the baseline value of the data to be processed is not within the preset range, the data to be processed is determined to be abnormal data.
[0108] Specifically, when the baseline value of the data to be processed is not within the preset range, that is, the deviation of the data to be processed from the dataset to be processed is not within the preset reasonable range, the data to be processed is determined to be abnormal data.
[0109] Optionally, the preset range can be a data range obtained from statistical calculations or a reasonable data range set based on human experience.
[0110] For example, the preset range can be determined by a first preset benchmark value and a second preset benchmark value. The first preset benchmark value is less than the second preset benchmark value. The first and second preset benchmark values can be calculated using statistical formulas or set based on human experience. Therefore, when the benchmark value of the data to be processed is less than the first preset benchmark value or greater than the second preset benchmark value, the data to be processed is determined to be abnormal data.
[0111] S503. If the baseline value of the data to be processed is within the preset range, determine that the data to be processed is not abnormal data.
[0112] Specifically, when the baseline value of the data to be processed is within a preset range, that is, when the deviation of the data to be processed is within a preset reasonable range, or when the fluctuation of the data to be processed conforms to the distribution characteristics of the dataset to be processed, the data to be processed is determined not to be abnormal data.
[0113] For example, when the baseline value of the data to be processed is between the first preset baseline value and the second preset baseline value, that is, the fluctuation range of the data to be processed is within a reasonable range, the data to be processed is determined to be non-abnormal data.
[0114] As can be seen, this embodiment analyzes the fluctuations of the data to be processed in the dataset to determine whether the data is abnormal, and thus determines whether there are anomalies in the dataset. Therefore, implementing this embodiment can enable timely anomaly detection and judgment on the stored dataset, thereby improving the accuracy and quality of the stored data. It avoids adverse effects caused by abnormal data during subsequent data analysis and decision-making.
[0115] In some embodiments, combined with Figure 5 ,like Figure 6 As shown, in the above S502, after determining that the data to be processed is abnormal data when the baseline value of the data to be processed is not within the preset range, the following steps are also included:
[0116] S601. Determine the baseline score of the dataset to be processed based on the baseline value of the abnormal data.
[0117] Among them, the basic score is positively correlated with the benchmark value.
[0118] For example, the difference between the baseline value of the outlier data and either a first preset baseline value or a second preset baseline value is determined. When the baseline value of the outlier data is less than the first preset baseline value, the baseline difference is determined by the difference between the baseline value of the outlier data and the first preset baseline value. When the baseline value of the outlier data is greater than the second preset baseline value, the baseline difference is determined by the difference between the baseline value of the outlier data and the second preset baseline value. Then, the baseline score of the dataset to be processed is determined using the baseline difference. The larger the baseline difference, the higher the degree of deviation of the outlier data; therefore, the larger the baseline score used to determine the outlier score of the dataset to be processed.
[0119] S602. Based on the base score, determine the outlier scores of the dataset to be processed.
[0120] Specifically, the baseline score is set based on the degree of deviation of the outlier data. Therefore, the baseline score represents the degree of data anomalousness of the outlier data in the dataset to be processed. Thus, the anomaly score of the dataset to be processed is determined based on the baseline score.
[0121] S603. If the anomaly score of the dataset to be processed is greater than the preset anomaly score, issue an early warning for the dataset to be processed.
[0122] Optionally, the preset anomaly score can be a preset condition set based on human experience. When the anomaly score of the dataset to be processed exceeds the preset condition, an alert is issued for the dataset to be processed. Conversely, if the anomaly score of the dataset to be processed is less than or equal to the preset condition, no alert is issued for the dataset to be processed.
[0123] As can be seen, this embodiment sets a basic score for the data to be processed based on the degree of deviation of the abnormal data, and then determines an anomaly score representing the abnormality of the data to be processed based on the basic score. By using the anomaly score and setting reasonable anomaly warning conditions to complete the early warning response to abnormal data, it helps managers to promptly detect and handle data anomalies based on anomaly warnings. This allows managers to take timely measures when data problems occur, preventing potential risks from escalating further, thereby ensuring the accuracy and integrity of stored data.
[0124] In some embodiments, combined with Figure 6 ,like Figure 7 As shown, in S602 above, the anomaly scores of the dataset to be processed are determined based on the base scores, including:
[0125] S701, Obtain the storage permission level for abnormal data.
[0126] Specifically, the base score of the dataset to be processed needs to be adjusted based on the storage permission level of the abnormal data. Therefore, it is necessary to obtain the storage permission level of the abnormal data.
[0127] S702. Determine the anomaly score of the dataset to be processed based on the storage permission level and basic score of the anomaly data.
[0128] Specifically, since the anomaly score of the dataset to be processed is determined by the base score, when adjusting the anomaly score of the dataset to be processed according to the storage permission level of the anomaly data, it is necessary to determine the anomaly score of the dataset to be processed based on the storage permission level of the anomaly data and the base score.
[0129] It is understandable that, since data with different storage permission levels have different levels of importance, in order to ensure that abnormal data with higher storage permissions receives more attention during the process of determining the anomaly score of the dataset to be processed based on the basic score, this embodiment of the application further adds an adjustment to the anomaly score based on the storage permission of the abnormal data, on the basis of determining the anomaly score through the basic score.
[0130] In some embodiments, combined with Figure 7 ,like Figure 8 As shown, in S702 above, the anomaly score of the dataset to be processed is determined based on the storage permission level and basic score of the anomaly data, including:
[0131] S801. If the storage permission level of abnormal data is lower than the preset permission level, the basic score will be determined as the abnormal score.
[0132] Specifically, when the storage permission level of abnormal data is lower than the preset permission level, that is, the storage permission of abnormal data is low, there is no need to adjust the basic score. Therefore, the basic score is determined as the abnormal score.
[0133] S802. When the storage permission level of abnormal data is greater than or equal to the preset permission level, determine the target adjustment coefficient according to the storage permission level of the abnormal data, and adjust the basic score according to the target adjustment coefficient to obtain the abnormal score.
[0134] Specifically, when the storage permission level of abnormal data is greater than or equal to the preset permission level, meaning the abnormal data has a high storage permission, the base score needs to be adjusted. Therefore, based on the storage permission level of the abnormal data, a target adjustment coefficient is determined for adjusting the base score, and the adjusted base score is then designated as the abnormal score.
[0135] It is understandable that adjusting the basic score based on the permission level of abnormal data allows for the development of reasonable abnormal data handling strategies tailored to the actual operation of the system. For example, data with higher permission levels is usually more critical, and if it becomes abnormal and is not properly handled, it may cause serious security problems. Therefore, this embodiment of the application adjusts the basic score based on the storage permissions of abnormal data, enabling administrators to focus on high-risk abnormal data (i.e., abnormal data with higher data storage permissions), ensuring the overall security of the data when performing anomaly analysis on the dataset to be processed.
[0136] In some embodiments, combined with Figure 8 ,like Figure 9 As shown, in S802 above, the target adjustment coefficient is determined based on the storage permission level of the abnormal data, specifically including:
[0137] S901. Determine the difference between the storage permission level of abnormal data and the preset permission level.
[0138] Specifically, when the storage permission level of abnormal data is higher than the preset permission level, the base score needs to be adjusted based on the specific differences between the abnormal data's storage permission level and the preset permission level. Therefore, it is necessary to determine the level difference between the abnormal data's storage permission level and the preset permission level.
[0139] Optionally, the preset permission level can be set according to the actual operation of each system when storing and detecting anomalies in multiple different data systems.
[0140] S902. The ratio between the grade difference and the preset grade difference is determined as the initial adjustment coefficient.
[0141] It is understandable that during the data lifecycle, factors such as the importance of the data and its environment may change, leading to alterations in the relationship between the data's storage permission level and the preset permission level. Therefore, this application embodiment calculates the difference between the storage permission level of abnormal data and the preset permission level in real time, and compares this difference with the preset level difference to form a ratio to determine an adjustment coefficient. This allows for dynamic adjustment of the anomaly score based on the actual state of the data. This provides greater flexibility in adapting to various changes in data during storage and anomaly data analysis.
[0142] S903. Obtain the amount of duplicate data in the dataset to be processed.
[0143] Specifically, the data to be processed includes data obtained from multiple different data systems. Therefore, there will be data duplication in the dataset. It is necessary to determine the amount of data duplication in the dataset so that the impact of duplicate data on the anomaly scores of the dataset can be removed later.
[0144] It is understandable that the anomaly score of the dataset to be processed aims to accurately reflect the deviation of abnormal data from the normal state in the dataset. However, duplicate data in the dataset may reduce the accuracy of the anomaly score and fail to truly reflect the anomalies in the dataset. Therefore, in this embodiment, after adjusting the basic score according to the storage permissions of the abnormal data, the analysis of duplicate data in the dataset to be processed is added to eliminate the impact of duplicate data on the anomaly score of the dataset to be processed.
[0145] S904. Determine the target adjustment factor based on the data repetition rate and the initial adjustment factor.
[0146] Specifically, the initial adjustment coefficients are adjusted based on the repetition of data in the dataset to be processed in order to determine the target adjustment coefficients.
[0147] Optionally, after adjusting the base score through the storage permissions of abnormal data, the analysis and processing of the amount of duplicate data in the dataset to be processed can be added. The initial adjustment coefficient for adjusting the base score can be corrected to determine the target adjustment coefficient for adjusting the base score, thereby eliminating the impact of duplicate data in the dataset to be processed on the abnormal scores of the dataset to be processed.
[0148] In some embodiments, combined with Figure 9 ,like Figure 10 As shown, in S904 above, the target adjustment coefficient is determined based on the data repetition rate and the initial adjustment coefficient, specifically including:
[0149] S1001. When the amount of data repetition is less than the preset amount of data repetition, or the difference between the amount of data repetition and the preset amount of data repetition is less than the preset difference, the initial adjustment coefficient is determined as the target adjustment coefficient.
[0150] Optionally, when the amount of data repetition is less than the preset amount of data repetition, or the difference between the amount of data repetition and the preset amount of data repetition is less than the preset difference, that is, the degree of data repetition in the dataset to be processed is within an acceptable range, therefore, there is no need to further adjust the initial adjustment coefficient of the basic score, so the initial adjustment coefficient is determined as the target adjustment coefficient.
[0151] S1002. When the amount of data repetition is greater than or equal to the preset amount of data repetition, and the difference between the amount of data repetition and the preset amount of data repetition is greater than or equal to the preset difference, determine the correction coefficient for correcting the initial adjustment coefficient, and correct the initial adjustment coefficient according to the correction coefficient to obtain the target adjustment coefficient.
[0152] Optionally, when the amount of data duplication in the dataset to be processed exceeds the preset condition, it indicates that the degree of data duplication in the current dataset to be processed will affect the accuracy of the anomaly score of the dataset to be processed. Therefore, by correcting the initial adjustment coefficient of the basic score of the dataset to be processed, the anomaly score of the dataset to be processed can be determined, the influence of duplicate data in the dataset to be processed on the anomaly score of the dataset to be processed can be eliminated, thereby improving the accuracy of anomaly analysis of the dataset to be processed.
[0153] In some embodiments, combined with Figure 10 In S1002 above, the correction coefficients for correcting the initial adjustment coefficients are determined, specifically including:
[0154] The correction coefficient is determined based on the difference between the data duplication rate and the preset data duplication rate, and the total amount of data in the dataset to be processed.
[0155] Among them, the correction coefficient is negatively correlated with the amount of data duplication.
[0156] Optionally, the percentage of repeated data can be obtained by dividing the difference between the repeated data amount and the preset repeated data amount by the total amount of data in the dataset to be processed. Furthermore, since the correction coefficient is negatively correlated with the repeated data amount, the correction coefficient can be determined by subtracting the percentage of repeated data from the total percentage.
[0157] As can be seen, this embodiment of the application performs anomaly detection on the stored data, and further determines the basic score of the dataset to be processed corresponding to the anomaly data based on the detected anomaly data. Then, the obtained basic score is adjusted from multiple aspects such as the permissions of the anomaly data and the amount of data duplication in the dataset to be processed, thereby improving the accuracy of anomaly data analysis and realizing statistical analysis of anomaly data in the overall data.
[0158] In some embodiments, combined with Figure 6 ,like Figure 11 As shown in S603 above, issuing an early warning for the dataset to be processed specifically includes:
[0159] S1101. Determine the difference between the abnormal score and the preset abnormal score.
[0160] Specifically, the warning level needs to be determined based on the relationship between the abnormal score difference and the preset warning score difference. Therefore, the abnormal score difference needs to be determined based on the specific difference between the abnormal score and the preset abnormal score.
[0161] S1102. Determine the warning level based on the difference between the abnormal score difference and the preset warning score difference.
[0162] Among them, the difference between the warning level and the difference between the abnormal score and the preset warning score is positively correlated.
[0163] Optionally, the warning level can be determined by judging whether the difference between the abnormal score difference and the preset warning score difference falls within a preset warning level range.
[0164] S1103. Issue warnings to the dataset to be processed based on the warning level.
[0165] Optionally, warning levels can be classified based on preset warning levels according to the determined warning levels, so as to realize hierarchical warning of anomalies in the dataset to be processed.
[0166] Understandably, issuing warnings based on different alert levels for the dataset to be processed can intuitively and accurately reflect any anomalies in the dataset. Furthermore, tiered warning systems can be implemented based on alert levels, allowing administrators to take appropriate countermeasures based on different alert levels, effectively improving the speed of processing abnormal data.
[0167] As can be seen, this embodiment of the application acquires the data to be processed from the dataset to be processed, performs anomaly detection on the data, and further determines the basic score for anomaly assessment of the dataset by analyzing the data to be processed and the dataset itself, based on the premise that the data to be processed is anomalous (i.e., there are anomalies in the dataset). To ensure the accuracy of the anomaly assessment of the dataset, this embodiment of the application analyzes multiple aspects of the dataset, including the storage permissions of the anomalous data and the amount of data duplication, and adjusts the basic score for anomaly assessment to determine the anomaly score of the dataset. This improves the comprehensiveness and accuracy of anomaly identification in the dataset, achieves accurate assessment of anomalies in the dataset, and realizes the assessment of data anomalies from a holistic perspective. Subsequently, this embodiment of the application further analyzes the anomaly score and preset anomaly score and classifies the warning levels to achieve hierarchical warnings for the dataset, improving the efficiency of management personnel in processing anomalous data.
[0168] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0169] This application embodiment can divide the device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0170] Figure 12A schematic diagram of an abnormal data determination device provided in an embodiment of this application is shown. Figure 12 As shown, the abnormal data determination device includes a communication unit 1201 and a processing unit 1202.
[0171] The communication unit 1201 is used to acquire data to be classified from multiple different data systems.
[0172] The processing unit 1202 is used to classify the dataset according to the attribute information of the data to be classified, and to determine at least one category of the dataset to be processed.
[0173] The processing unit 1202 is also used to obtain the data distribution characteristics of the dataset to be processed for each category of dataset.
[0174] The dataset to be processed includes data of the same business type from multiple different data systems.
[0175] The processing unit 1202 is also used to determine whether the data to be processed is abnormal data based on the data to be processed in the dataset to be processed and the data distribution characteristics of the dataset to be processed.
[0176] Optionally, the processing unit 1202 is specifically used to: determine a baseline value for the data to be processed based on the data distribution characteristics of the data to be processed and the dataset to be processed. The baseline value represents the business accuracy and / or numerical accuracy of the data to be processed. If the baseline value of the data to be processed is not within a preset range, the data to be processed is determined to be abnormal data. If the baseline value of the data to be processed is within a preset range, the data to be processed is determined not to be abnormal data.
[0177] Optionally, the processing unit 1202 is specifically used for: determining a baseline score for the dataset to be processed based on the baseline value of the abnormal data. The baseline score is positively correlated with the baseline value. Based on the baseline score, determining the abnormal score of the dataset to be processed. If the abnormal score of the dataset to be processed is greater than a preset abnormal score, issuing an early warning for the dataset to be processed.
[0178] Optionally, processing unit 1202 is specifically used for: obtaining the storage permission level of abnormal data; and determining the anomaly score of the dataset to be processed based on the storage permission level and the base score of the abnormal data.
[0179] Optionally, the processing unit 1202 is specifically used to: determine the basic score as an abnormal score when the storage permission level of the abnormal data is less than the preset permission level; and determine a target adjustment coefficient based on the storage permission level of the abnormal data when the storage permission level of the abnormal data is greater than or equal to the preset permission level, and adjust the basic score according to the target adjustment coefficient to obtain the abnormal score.
[0180] Optionally, the processing unit 1202 is specifically used for: determining the level difference between the storage permission level of the abnormal data and the preset permission level; determining the ratio between the level difference and the preset level difference as the initial adjustment coefficient; obtaining the amount of data duplication in the dataset to be processed; and determining the target adjustment coefficient based on the amount of data duplication and the initial adjustment coefficient.
[0181] Optionally, the processing unit 1202 is specifically configured to: determine an initial adjustment coefficient as a target adjustment coefficient when the amount of data repetition is less than a preset amount of data repetition, or when the difference in quantity between the amount of data repetition and the preset amount of data repetition is less than a preset difference in quantity; and determine a correction coefficient to adjust the initial adjustment coefficient when the amount of data repetition is greater than or equal to the preset amount of data repetition, and when the difference in quantity between the amount of data repetition and the preset amount of data repetition is greater than or equal to a preset difference in quantity, and adjust the initial adjustment coefficient according to the correction coefficient to obtain the target adjustment coefficient.
[0182] Optionally, the processing unit 1202 is specifically used to: determine a correction coefficient based on the difference between the data repetition rate and the preset data repetition rate, and the total amount of data in the dataset to be processed. The correction coefficient is negatively correlated with the data repetition rate.
[0183] Optionally, the processing unit 1202 is specifically used for: determining the difference between the anomaly score and a preset anomaly score; determining the warning level based on the difference between the anomaly score difference and the preset warning score difference; wherein the warning level is positively correlated with the difference between the anomaly score difference and the preset warning score difference; and issuing a warning to the dataset to be processed based on the warning level.
[0184] This application also provides a computer-readable storage medium, which includes computer-executable instructions. When the computer-executable instructions are run on a computer, the computer performs the abnormal data determination method provided in the above embodiments.
[0185] This application also provides a computer program that can be directly loaded into a memory and contains software code. After being loaded and executed by a computer, the computer program can implement the abnormal data determination method provided in the above embodiments.
[0186] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this application can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer.
[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0188] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and other division methods may exist in actual implementation. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate; components shown as units may be one physical unit or multiple physical units, i.e., they may be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0189] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to general technology, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0190] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for identifying abnormal data, characterized in that, include: Obtain the data to be classified from multiple different data systems; Classify the data to be classified based on its attribute information to determine at least one category of the dataset to be processed; For each category of dataset to be processed, obtain the data distribution characteristics of the dataset to be processed; the dataset to be processed includes data of the same business type from multiple different data systems; Based on the data to be processed in the dataset and the data distribution characteristics of the dataset, determine whether the data to be processed is abnormal data.
2. The method according to claim 1, characterized in that, The step of determining whether the data to be processed is abnormal data based on the data to be processed in the dataset and the data distribution characteristics of the dataset includes: Based on the data distribution characteristics of the data to be processed and the dataset to be processed, a baseline value for the data to be processed is determined; the baseline value is used to represent the business accuracy and / or numerical accuracy of the data to be processed. If the baseline value of the data to be processed is not within a preset range, the data to be processed is determined to be abnormal data. If the baseline value of the data to be processed is within the preset range, it is determined that the data to be processed is not the abnormal data.
3. The method according to claim 2, characterized in that, After determining that the data to be processed is abnormal data when the baseline value of the data to be processed is not within a preset range, the method further includes: Based on the baseline value of the abnormal data, a basic score is determined for the dataset to be processed; the basic score is positively correlated with the baseline value. Based on the base score, determine the anomaly score of the dataset to be processed; If the anomaly score of the dataset to be processed is greater than the preset anomaly score, an early warning will be issued for the dataset to be processed.
4. The method according to claim 3, characterized in that, The step of determining the anomaly score of the dataset to be processed based on the base score includes: Obtain the storage permission level of the abnormal data; The anomaly score of the dataset to be processed is determined based on the storage permission level of the abnormal data and the basic score.
5. The method according to claim 4, characterized in that, The step of determining the anomaly score of the dataset to be processed based on the storage permission level of the anomaly data and the basic score includes: If the storage permission level of the abnormal data is lower than the preset permission level, the basic score will be determined as the abnormal score. If the storage permission level of the abnormal data is greater than or equal to the preset permission level, a target adjustment coefficient is determined based on the storage permission level of the abnormal data, and the base score is adjusted based on the target adjustment coefficient to obtain the abnormal score.
6. The method according to claim 5, characterized in that, The step of determining the target adjustment coefficient based on the storage permission level of the abnormal data includes: Determine the level difference between the storage permission level of the abnormal data and the preset permission level; The ratio between the stated grade difference and the preset grade difference is determined as the initial adjustment coefficient; Obtain the amount of duplicate data in the dataset to be processed; The target adjustment coefficient is determined based on the data repetition rate and the initial adjustment coefficient.
7. The method according to claim 6, characterized in that, The step of determining the target adjustment coefficient based on the data repetition rate and the initial adjustment coefficient includes: If the amount of data repetition is less than the preset amount of data repetition, or if the difference between the amount of data repetition and the preset amount of data repetition is less than the preset difference, the initial adjustment coefficient is determined to be the target adjustment coefficient. If the amount of data repetition is greater than or equal to a preset amount of data repetition, and the difference between the amount of data repetition and the preset amount of data repetition is greater than or equal to a preset difference, a correction coefficient is determined to correct the initial adjustment coefficient, and the initial adjustment coefficient is corrected according to the correction coefficient to obtain the target adjustment coefficient.
8. The method according to claim 7, characterized in that, The determination of the correction coefficient for correcting the initial adjustment coefficient includes: The correction coefficient is determined based on the difference between the data repetition amount and the preset data repetition amount, and the total amount of data in the dataset to be processed; the correction coefficient is negatively correlated with the data repetition amount.
9. The method according to claim 3, characterized in that, The method of issuing early warnings for the dataset to be processed includes: The difference between the abnormal score and the preset abnormal score is determined. The warning level is determined based on the difference between the abnormal score difference and the preset warning score difference; the warning level is positively correlated with the difference between the abnormal score difference and the preset warning score difference. The dataset to be processed will be given an early warning based on the warning level.
10. An abnormal data identification system, characterized in that, include: Acquisition module and storage module; The acquisition module is used to acquire data to be classified from multiple different data systems; The storage module is used to classify the data to be classified according to the attribute information of the data to be classified, and to determine at least one category of the dataset to be processed. The storage module is also used to obtain the data distribution characteristics of the dataset to be processed for each category; The storage module is further configured to determine whether the data to be processed is abnormal data based on the data to be processed in the dataset to be processed and the data distribution characteristics of the dataset to be processed.