Data anomaly detection and analysis method and device, electronic equipment and storage medium

By acquiring the data set and its attribute description information, and determining the abnormality index and entity data using pre-set methods, the problem of low noise data processing efficiency in the prior art is solved, and the efficiency and accuracy of abnormal data detection are improved.

CN119088646APending Publication Date: 2024-12-06BEIJING YOUTEJIE INFORMATION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411271304.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art is difficult to eliminate noise data in data abnormality detection, and the processing volume is large and the efficiency is low, resulting in insufficient efficiency and accuracy of abnormal data detection.

Method used

By obtaining the data set to be detected and its data attribute description information, the current abnormality index data is determined using the pre-set exception data detection method, and the abnormal entity and entity data are determined based on the service level target indicators, and feedback is output to the user.

Benefits of technology

It improves the efficiency and accuracy of abnormal data detection, reduces the workload, and solves the problem that it is difficult to eliminate noise data in large amounts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088646A_ABST
    Figure CN119088646A_ABST
Patent Text Reader

Abstract

The invention discloses a data anomaly detection and analysis method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a data set to be subjected to anomaly detection and analysis, and obtaining data attribute description information corresponding to multiple pieces of data to be subjected to anomaly detection and analysis in the data set to be subjected to anomaly detection and analysis; according to the data attribute description information, determining at least one piece of current abnormal index data through a preset abnormal data detection method; acquiring a machine entity type and a process entity type respectively corresponding to each piece of current abnormal index data, and determining an abnormal entity and abnormal entity data through a preset service level target index; and outputting and feeding back the current abnormal index data, the abnormal entities and the abnormal entity data to a user. The problems that the abnormal data detection data volume is large, noise data is difficult to remove and the effectiveness is poor are solved. The efficiency and accuracy of abnormal data detection are improved, and the workload of abnormal data detection is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data anomaly detection and analysis method, device, electronic equipment and storage medium. Background Art

[0002] With the informatization of data, more and more data has appeared, including normal data or abnormal data. Specifically, abnormal data will affect the operation of equipment, so the judgment of data anomalies is also crucial.

[0003] In addition, sometimes the data has not yet reported an error, but there is a trend of an error. It is hoped that it can be detected before the error is reported, and an alarm notification can be issued to inform the operation and maintenance personnel of the relevant information so that the operation and maintenance personnel can analyze and process it, which can reduce the harm caused by abnormal data.

[0004] In the process of realizing the present invention, the inventors found that the prior art has the following defects: At present, for data anomaly detection, there is a baseline for comparison, for example, comparison with historical data, if it exceeds a certain percentage, it indicates an anomaly, but this detection technology is difficult to eliminate noise data, the amount of data to be processed is relatively large, and the efficiency is low. Summary of the invention

[0005] The present invention provides a data anomaly detection and analysis method, device, electronic device and storage medium to improve the efficiency and accuracy of abnormal data detection.

[0006] According to one aspect of the present invention, a data anomaly detection and analysis method is provided, which includes:

[0007] Acquire a data set to be analyzed for anomaly detection, and acquire data attribute description information corresponding to a plurality of data to be analyzed for anomaly detection in the data set to be analyzed for anomaly detection;

[0008] According to the data attribute description information, at least one current abnormal indicator data is determined by a preset abnormal data detection method;

[0009] Obtaining the machine entity type and process entity type corresponding to each current abnormal indicator data, and determining the abnormal entity and abnormal entity data through the preset service level target indicator;

[0010] The current abnormal indicator data, abnormal entity and abnormal entity data are output and fed back to the user.

[0011] According to another aspect of the present invention, a data anomaly detection and analysis device is provided, comprising:

[0012] A data attribute description information acquisition module is used to acquire a data set to be analyzed for anomaly detection, and to acquire data attribute description information corresponding to a plurality of data to be analyzed for anomaly detection in the data set to be analyzed for anomaly detection;

[0013] A current abnormal indicator data determination module, used to determine at least one current abnormal indicator data according to the data attribute description information by a preset abnormal data detection method;

[0014] The abnormal entity and abnormal entity data determination module is used to obtain the machine entity type and process entity type corresponding to each current abnormal indicator data, and determine the abnormal entity and abnormal entity data through the preset service level target indicator;

[0015] The output feedback module is used to output and feedback the current abnormal indicator data, abnormal entities and abnormal entity data to the user.

[0016] According to another aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data anomaly detection and analysis method described in any embodiment of the present invention is implemented.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data anomaly detection and analysis method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention obtains a data set to be detected and analyzed for anomalies, and obtains data attribute description information corresponding to multiple data to be detected and analyzed for anomalies in the data set to be detected and analyzed for anomalies; according to the data attribute description information, at least one current abnormal index data is determined by a preset abnormal data detection method; the machine entity type and process entity type corresponding to each current abnormal index data are obtained, and the abnormal entity and abnormal entity data are determined by a preset service level target indicator; each current abnormal index data, abnormal entity and abnormal entity data are output and fed back to the user. The problem of large data volume, difficulty in eliminating noise data and poor effectiveness in abnormal data detection is solved. The efficiency and accuracy of abnormal data detection are improved, and the workload of abnormal data detection is reduced.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flow chart of a data anomaly detection and analysis method provided according to Embodiment 1 of the present invention;

[0022] Figure 2 is a structural schematic diagram of a data anomaly detection and analysis device provided according to Embodiment 2 of the present invention;

[0023] Figure 3 It is a structural schematic diagram of an electronic device provided according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "target", "current", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] Embodiment 1

[0027] Figure 1 A flowchart of a data anomaly detection and analysis method provided in Example 1 of the present invention. This embodiment is applicable to the situation of time consumption prediction of distributed task resources. The method can be executed by a data anomaly detection and analysis device, and the data anomaly detection and analysis device can be implemented in the form of hardware and / or software.

[0028] Correspondingly, such as Figure 1 As shown, the method includes:

[0029] S110, obtaining a data set to be analyzed for anomaly detection, and obtaining data attribute description information corresponding to a plurality of data to be analyzed for anomaly detection in the data set to be analyzed for anomaly detection.

[0030] The data set to be analyzed for anomaly detection may be a data set that needs to be analyzed for anomaly detection, and the data set to be analyzed for anomaly detection may include a plurality of data to be analyzed for anomaly detection. The data attribute description information may be information describing specific attributes of the data.

[0031] Specifically, the data attribute description information includes entity type, data indicators and entity number; among them, entity type includes service interface entity type, machine entity type and process entity type; the data indicator corresponding to the service interface entity type is time consumption; the data indicators corresponding to the machine entity type are central processing unit memory, disk memory and storage memory; the data indicators corresponding to the process entity type are process central processing unit memory and process storage memory.

[0032] In this embodiment, the data attribute description information may include entity type, data index and entity number. Specifically, different entity types correspond to different data indexes. Specifically, the data index corresponding to the service interface entity type may be time consumption; the data index corresponding to the machine entity type may be CPU memory, disk memory and storage memory; the data index corresponding to the process entity type may be process CPU memory and process storage memory.

[0033] S120: Determine at least one current abnormal indicator data according to the data attribute description information by using a preset abnormal data detection method.

[0034] The current abnormal indicator data may be data belonging to abnormal indicators in the data of the service interface entity type.

[0035] Optionally, the determining of at least one current abnormal indicator data according to the data attribute description information through a preset abnormal data detection method includes: obtaining data whose entity type is a service interface entity type in each of the data to be detected and analyzed for abnormality through a preset abnormal data detection method; calculating the average time consumption, the standard deviation of the time consumption, the average overall time consumption, and the standard deviation of the overall time consumption according to the time consumption and the entity number in the service interface entity type; calculating the allowable time range according to the average time consumption and the standard deviation of the time consumption, and comparing the time consumption with the allowable time range to obtain first abnormal data; calculating the allowable total time range according to the average overall time consumption and the standard deviation of the overall time consumption, and comparing the time consumption with the allowable total time range to obtain second abnormal data; determining at least one current abnormal indicator data according to the first abnormal data and the second abnormal data.

[0036] In this embodiment, first, it is necessary to obtain data of service interface entity type from each data to be analyzed for abnormality detection. As shown in Table 1 below, data of 10 service interface entity types are obtained. In Table 1, two groups of data with entity numbers 987 and 654 can be included, and the specific time consumption corresponding to each group of data can be obtained.

[0037] Further, the average calculation time, the standard deviation of the time, the average overall time and the standard deviation of the overall time can be calculated respectively to determine the first abnormal data and the second abnormal data, thereby determining the current abnormal indicator data.

[0038] Table 1

[0039]

[0040] Specifically, the method of calculating the time consumption average, time consumption standard deviation, overall time consumption average and overall time consumption standard deviation based on the time consumption and entity number in the service interface entity type includes: grouping the same numbers according to the entity numbers to obtain at least one group of the same entity numbers; wherein each group of the same entity numbers includes at least one same entity number; calculating the time consumption average and time consumption standard deviation corresponding to each group of the same entity numbers based on the time consumption in each group of the same entity numbers; and calculating the overall time consumption average and overall time consumption standard deviation based on the time consumption of all data in the service interface entity type.

[0041] Continuing with the previous example, the entity numbers corresponding to the data in Table 1 need to be grouped with the same number, and a group of 987 entity numbers (including 4 data) and a group of 654 entity numbers (including 6 data) can be obtained.

[0042] For the group of 987 entity numbers, the average time consumption can be calculated as The standard deviation of the time consumption is Among them, x i The time consumption corresponding to different data.

[0043] For the group of 654 entity numbers, the average time consumption can be calculated as The standard deviation of the time consumption is Among them, x i The time consumption corresponding to different data.

[0044] The specific calculation for the overall average time consumption is: The specific calculation of the overall time standard deviation is:

[0045] Optionally, the method of calculating the allowable time range based on the average time consumption and the standard deviation of the time consumption, and comparing the time consumption with the allowable time range to obtain the first abnormal data, includes: adding the average time consumption corresponding to each group of identical entity numbers to the standard deviation of the time consumption to calculate the maximum allowable time consumption, and subtracting the standard deviation of the time consumption from the average time consumption to calculate the minimum allowable time consumption; generating the allowable time range corresponding to each group of identical entity numbers based on the maximum allowable time consumption and the minimum allowable time consumption; judging whether the allowable time range is met based on the time consumption corresponding to each data in the service interface entity type, and if so, determining it as the first normal data; if not, determining it as the first abnormal data.

[0046] Continuing with the previous example, for the group with entity number 987, since the average time consumption is 2ms and the standard deviation of time consumption is 0.7071, we can calculate that the maximum allowed time consumption is 2.7071 and the minimum allowed time consumption is 1.2929. Therefore, we can determine that the allowed range of time consumption for the group with entity number 987 is [1.2929, 2.7071].

[0047] Similarly, it can be determined that the allowed range of time consumption for the group with entity number 654 is [0.7066, 3.6266].

[0048] Furthermore, it is necessary to judge whether the allowed time range is met based on the time consumption corresponding to each data in the service interface entity type. For the first data of entity number 987, the time consumption is 2, which meets the allowed time range [1.2929, 2.7071], then the data is the first normal data. For the first data of entity number 654, the time consumption is 5, which does not meet the allowed time range [0.7066, 3.6266], then the data is the first abnormal data. Similarly, the situation of the remaining 8 data can be judged, which will not be repeated here. After calculation, for the fourth data of entity number 987, the time consumption is 3, which does not meet the allowed time range [1.2929, 2.7071], then the data is the first abnormal data.

[0049] Optionally, the allowable range of total time consumption is calculated based on the average total time consumption and the standard deviation of the total time consumption, and the time consumption is compared with the allowable range of total time consumption to obtain the second abnormal data, including: adding the average total time consumption to the standard deviation of the total time consumption to calculate the maximum allowable total time consumption, and subtracting the standard deviation of the total time consumption from the average total time consumption to calculate the minimum allowable total time consumption; generating the allowable range of total time consumption based on the maximum allowable total time consumption and the minimum allowable total time consumption; judging whether the allowable range of total time consumption is met based on the time consumption corresponding to each data in the service interface entity type, and if so, determining it as the second normal data; if not, determining it as the second abnormal data.

[0050] Continuing with the previous example, since the calculated average total time consumption is 2.1ms and the total time consumption standard deviation is 1.22, we can calculate that the maximum allowable total time consumption is 3.32ms and the minimum allowable total time consumption is 0.88ms. Further, we can determine that the allowable range of the total time consumption is [0.88, 3.32].

[0051] Furthermore, the time consumption corresponding to each data in the service interface entity type can be used to determine whether the total time consumption range is met. If it is met, it is determined as the second normal data; if not, it is determined as the second abnormal data. It can be determined that for the first data of entity number 654, the time consumption is 5, which does not meet the total time consumption range [0.88, 3.32], then the data is the second abnormal data, and the remaining 9 data are all the second normal data.

[0052] Correspondingly, current abnormality indicator data is determined according to the first abnormality data and the second abnormality data, including 2 current abnormality indicator data.

[0053] In addition, for each group of data with the same entity number, if a preset number of abnormal data continuously appears, an alarm can be processed, and other entity types corresponding to the entity numbers corresponding to the abnormal data can also be obtained. For example, the machine entity type and the process entity type can be obtained.

[0054] S130: Obtain the machine entity type and process entity type corresponding to each current abnormal indicator data, and determine the abnormal entity and abnormal entity data through the preset service level target indicator.

[0055] The corresponding machine entity type and process entity type can be obtained through the entity number. Specifically, the machine entity type can correspond to a machine service level target indicator, which includes the standard indicator requirements for CPU memory, disk memory and storage memory. In addition, the process entity type can correspond to a process service level target indicator, which includes the standard indicator requirements for process CPU memory and process storage memory.

[0056] Optionally, the obtaining of machine entity types and process entity types respectively corresponding to each current abnormal indicator data, and determining abnormal entities and abnormal entity data through pre-set service level target indicators, includes: obtaining machine entity types and process entity types respectively corresponding to each current abnormal indicator data, and obtaining machine service level target indicators and process service level target indicators; respectively judging whether the central processing unit memory, disk memory and storage memory corresponding to the machine entity type meet the machine service level target indicators, and if so, determining the target current abnormal indicator data as normal entity data; if not, determining the target current abnormal indicator data as abnormal entity data; respectively judging whether the process central processing unit memory and process storage memory corresponding to the process entity type meet the process service level target indicators, and if so, determining the target current abnormal indicator data as normal entity data; if not, determining the target current abnormal indicator data as abnormal entity data; determining the abnormal entity according to the entity types respectively corresponding to the abnormal entity data.

[0057] Specifically, the machine service level objective index may include a CPU memory standard index, a disk memory standard index, and a storage memory standard index. The process service level objective index may include a process CPU memory standard index and a process storage memory standard index.

[0058] In this embodiment, it is necessary to determine whether the CPU memory, disk memory and storage memory corresponding to the machine entity type meet the machine service level target indicators; and it is necessary to determine whether the process CPU memory and process storage memory corresponding to the process entity type meet the process service level target indicators; thereby, abnormal entity data and abnormal entities can be determined.

[0059] S140: Output the current abnormal indicator data, abnormal entity and abnormal entity data to the user for feedback.

[0060] In this embodiment, each current abnormal indicator data, abnormal entity and abnormal entity data may be output as feedback to the user in a visual manner. For example, the feedback processing operation may be performed in the form of a table.

[0061] The technical solution of the embodiment of the present invention obtains a data set to be detected and analyzed for anomalies, and obtains data attribute description information corresponding to multiple data to be detected and analyzed for anomalies in the data set to be detected and analyzed for anomalies; according to the data attribute description information, at least one current abnormal index data is determined by a preset abnormal data detection method; the machine entity type and process entity type corresponding to each current abnormal index data are obtained, and the abnormal entity and abnormal entity data are determined by a preset service level target indicator; each current abnormal index data, abnormal entity and abnormal entity data are output and fed back to the user. The problem of large data volume, difficulty in eliminating noise data and poor effectiveness in abnormal data detection is solved. The efficiency and accuracy of abnormal data detection are improved, and the workload of abnormal data detection is reduced.

[0062] Embodiment 2

[0063] Figure 2 This is a schematic diagram of the structure of a data anomaly detection and analysis device provided in the second embodiment of the present invention. The data anomaly detection and analysis device provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or a server to implement a data anomaly detection and analysis method in the embodiment of the present invention. Figure 2 As shown, the device includes: a data attribute description information acquisition module 210, a current abnormal indicator data determination module 220, an abnormal entity and abnormal entity data determination module 230 and an output feedback module 240.

[0064] The data attribute description information acquisition module 210 is used to acquire the data set to be analyzed for anomaly detection, and to acquire data attribute description information corresponding to a plurality of data to be analyzed for anomaly detection in the data set to be analyzed for anomaly detection;

[0065] The current abnormal indicator data determination module 220 is used to determine at least one current abnormal indicator data according to the data attribute description information by a preset abnormal data detection method;

[0066] The abnormal entity and abnormal entity data determination module 230 is used to obtain the machine entity type and process entity type corresponding to each current abnormal indicator data, and determine the abnormal entity and abnormal entity data according to the preset service level target indicator;

[0067] The output feedback module 240 is used to output and feedback the current abnormal indicator data, abnormal entities and abnormal entity data to the user.

[0068] The technical solution of the embodiment of the present invention obtains a data set to be detected and analyzed for anomalies, and obtains data attribute description information corresponding to multiple data to be detected and analyzed for anomalies in the data set to be detected and analyzed for anomalies; according to the data attribute description information, at least one current abnormal index data is determined by a preset abnormal data detection method; the machine entity type and process entity type corresponding to each current abnormal index data are obtained, and the abnormal entity and abnormal entity data are determined by a preset service level target indicator; each current abnormal index data, abnormal entity and abnormal entity data are output and fed back to the user. The problem of large data volume, difficulty in eliminating noise data and poor effectiveness in abnormal data detection is solved. The efficiency and accuracy of abnormal data detection are improved, and the workload of abnormal data detection is reduced.

[0069] Based on the above embodiments, the data attribute description information includes entity type, data indicators and entity number; wherein the entity type includes service interface entity type, machine entity type and process entity type; the data indicator corresponding to the service interface entity type is time consumption; the data indicators corresponding to the machine entity type are central processing unit memory, disk memory and storage memory; the data indicators corresponding to the process entity type are process central processing unit memory and process storage memory.

[0070] On the basis of the above embodiments, the current abnormal indicator data determination module 220 includes: a data acquisition unit for service interface entity type, which is used to obtain data of entity type being service interface entity type in each of the data to be detected and analyzed for abnormality through a preset abnormal data detection method; a time consumption average, time consumption standard deviation, overall time consumption average and overall time consumption standard deviation calculation unit, which is used to calculate the time consumption average, time consumption standard deviation, overall time consumption average and overall time consumption standard deviation according to the time consumption and entity number in the service interface entity type; a first abnormal data determination unit, which is used to calculate the allowable time range according to the time consumption average and the time consumption standard deviation, and compare the time consumption with the allowable time range to obtain first abnormal data; a second abnormal data determination unit, which is used to calculate the allowable total time range according to the overall time consumption average and the overall time consumption standard deviation, and compare the time consumption with the allowable total time range to obtain second abnormal data; a current abnormal indicator data determination unit, which is used to determine at least one current abnormal indicator data according to the first abnormal data and the second abnormal data.

[0071] On the basis of the above embodiments, the time consumption average, time consumption standard deviation, overall time consumption average and overall time consumption standard deviation calculation unit can be specifically used to: group the same number according to the entity number to obtain at least one group of the same entity number; wherein each group of the same entity number includes at least one same entity number; calculate the time consumption average and time consumption standard deviation corresponding to each group of the same entity number according to the time consumption in each group of the same entity number; calculate the overall time consumption average and the overall time consumption standard deviation according to the time consumption of all data in the service interface entity type.

[0072] On the basis of the above embodiments, the first abnormal data determination unit can be specifically used to: add the average time consumption corresponding to each group of identical entity numbers to the standard deviation of the time consumption to calculate the maximum allowable time consumption, and subtract the standard deviation of the time consumption from the average time consumption to calculate the minimum allowable time consumption; generate the allowable time consumption range corresponding to each group of identical entity numbers according to the maximum allowable time consumption and the minimum allowable time consumption; determine whether the allowable time consumption range is met according to the time consumption corresponding to each data in the service interface entity type, and if so, determine it as the first normal data; if not, determine it as the first abnormal data.

[0073] On the basis of the above embodiments, the second abnormal data determination unit can be specifically used to: add the average total time consumption to the standard deviation of the overall time consumption to calculate the maximum allowable total time consumption, and subtract the standard deviation of the total time consumption from the average total time consumption to calculate the minimum allowable total time consumption; generate the allowable range of total time consumption according to the maximum allowable total time consumption and the minimum allowable total time consumption; judge whether the allowable range of total time consumption is met according to the time consumption corresponding to each data in the service interface entity type, and if so, determine it as the second normal data; if not, determine it as the second abnormal data.

[0074] On the basis of the above embodiments, the abnormal entity and abnormal entity data determination module 230 can be specifically used to: obtain the machine entity type and process entity type corresponding to each current abnormal indicator data, and obtain the machine service level target indicator and the process service level target indicator; respectively determine whether the central processing unit memory, disk memory and storage memory corresponding to the machine entity type meet the machine service level target indicator, if so, determine the target current abnormal indicator data as normal entity data; if not, determine the target current abnormal indicator data as abnormal entity data; respectively determine whether the process central processing unit memory and process storage memory corresponding to the process entity type meet the process service level target indicator, if so, determine the target current abnormal indicator data as normal entity data; if not, determine the target current abnormal indicator data as abnormal entity data; determine the abnormal entity according to the entity types corresponding to the abnormal entity data.

[0075] The data anomaly detection and analysis device provided in the embodiment of the present invention can execute the data anomaly detection and analysis method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0076] Embodiment 3

[0077] Figure 3 A schematic diagram of the structure of an electronic device 10 that can be used to implement the third embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0078] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0079] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0080] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data anomaly detection and analysis method.

[0081] In some embodiments, the data anomaly detection analysis method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data anomaly detection analysis method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data anomaly detection analysis method in any other appropriate manner (e.g., by means of firmware).

[0082] The method comprises: obtaining a data set to be detected and analyzed for anomalies, and obtaining data attribute description information corresponding to a plurality of data to be detected and analyzed for anomalies in the data set to be detected and analyzed for anomalies; determining at least one current anomaly indicator data according to the data attribute description information by a preset anomaly data detection method; obtaining a machine entity type and a process entity type respectively corresponding to each current anomaly indicator data, and determining an abnormal entity and abnormal entity data by a preset service level target indicator; and outputting the current anomaly indicator data, abnormal entity and abnormal entity data for feedback to a user.

[0083] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0084] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0085] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0086] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0087] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0088] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0089] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0090] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0091] Embodiment 4

[0092] Embodiment 4 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute a data anomaly detection and analysis method when executed by a computer processor, the method comprising: obtaining a data set to be detected and analyzed for anomaly, and obtaining data attribute description information corresponding to multiple data to be detected and analyzed for anomaly in the data set to be detected and analyzed for anomaly; determining at least one current anomaly indicator data according to the data attribute description information through a preset anomaly data detection method; obtaining a machine entity type and a process entity type corresponding to each current anomaly indicator data, and determining an abnormal entity and abnormal entity data through a preset service level target indicator; and outputting the current anomaly indicator data, abnormal entity and abnormal entity data for feedback to a user.

[0093] Of course, the computer-readable storage medium provided in the embodiment of the present invention has computer-executable instructions that are not limited to the method operations described above, and can also execute related operations in the data anomaly detection and analysis method provided in any embodiment of the present invention.

[0094] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0095] It is worth noting that in the embodiment of the above-mentioned data anomaly detection and analysis device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0096] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data anomaly detection and analysis method, characterized in that: include: Acquire a data set to be analyzed for anomaly detection, and acquire data attribute description information corresponding to a plurality of data to be analyzed for anomaly detection in the data set to be analyzed for anomaly detection; According to the data attribute description information, at least one current abnormal indicator data is determined by a preset abnormal data detection method; Obtaining the machine entity type and process entity type corresponding to each current abnormal indicator data, and determining the abnormal entity and abnormal entity data through the preset service level target indicator; The current abnormal indicator data, abnormal entity and abnormal entity data are output and fed back to the user.

2. The method according to claim 1, characterized in that The data attribute description information includes entity type, data index and entity number; Among them, entity types include service interface entity types, machine entity types and process entity types; The data indicator corresponding to the service interface entity type is time consumption; The data indicators corresponding to the machine entity type are CPU memory, disk memory, and storage memory; The data indicators corresponding to the process entity type are process CPU memory and process storage memory.

3. The method according to claim 2, characterized in that The step of determining at least one current abnormal indicator data according to the data attribute description information by a preset abnormal data detection method includes: By using a preset abnormal data detection method, in each of the data to be detected and analyzed for abnormality, data whose entity type is a service interface entity type is obtained; According to the time consumption and entity number in the service interface entity type, calculate the average time consumption, the standard deviation of the time consumption, the average overall time consumption and the standard deviation of the overall time consumption; Calculating an allowable range of time consumption according to the average time consumption and the standard deviation of time consumption, and comparing the time consumption with the allowable range of time consumption to obtain first abnormal data; Calculate the total time consumption allowable range according to the total time consumption average and the total time consumption standard deviation, and compare the time consumption with the total time consumption allowable range to obtain second abnormal data; At least one current abnormality indicator data is determined according to the first abnormality data and the second abnormality data.

4. The method according to claim 3, characterized in that The method of calculating the average time consumption, the standard deviation of time consumption, the average time consumption and the standard deviation of overall time consumption according to the time consumption and the entity number in the service interface entity type includes: Grouping the same numbers according to the entity numbers to obtain at least one group of the same entity numbers; Wherein, each group of identical entity numbers includes at least one identical entity number; According to the time consumption in each group of the same entity number, the average time consumption and the standard deviation of the time consumption corresponding to each group of the same entity number are calculated; According to the time consumption of all data in the service interface entity type, the overall time consumption average and the overall time consumption standard deviation are calculated.

5. The method according to claim 4, characterized in that The method of calculating the allowable time range according to the average time and the standard deviation of the time, and comparing the time with the allowable time range to obtain the first abnormal data includes: The average time consumption corresponding to each group of the same entity number is added to the standard deviation of the time consumption to calculate the maximum allowable time consumption, and the standard deviation of the time consumption is subtracted from the average time consumption to calculate the minimum allowable time consumption; According to the maximum allowed time consumption and the minimum allowed time consumption, the allowed time consumption ranges corresponding to each group of the same entity number are generated; According to the time consumption corresponding to each data in the service interface entity type, it is determined whether the time consumption meets the allowed range. If it does, it is determined as the first normal data; if it does not, it is determined as the first abnormal data.

6. The method according to claim 5, characterized in that The method of calculating the allowable range of the total time consumption according to the average of the total time consumption and the standard deviation of the total time consumption, and comparing the time consumption with the allowable range of the total time consumption to obtain the second abnormal data includes: Add the total time average to the total time standard deviation to calculate the maximum allowable total time, and subtract the total time standard deviation from the total time average to calculate the minimum allowable total time; Generate the allowed range of total time consumption according to the maximum allowed value of total time consumption and the minimum allowed value of total time consumption; According to the time consumption corresponding to each data in the service interface entity type, it is judged whether the total time consumption meets the allowed range. If it does, it is determined as the second normal data; if not, it is determined as the second abnormal data.

7. The method according to claim 6, characterized in that The obtaining of the machine entity type and the process entity type respectively corresponding to each current abnormal indicator data, and determining the abnormal entity and abnormal entity data by using the preset service level target indicator, includes: Obtaining the machine entity type and process entity type corresponding to each current abnormal indicator data, and obtaining the machine service level target indicator and process service level target indicator; Determine respectively whether the CPU memory, disk memory and storage memory corresponding to the machine entity type meet the machine service level target indicators. If so, determine the target current abnormal indicator data as normal entity data; if not, determine the target current abnormal indicator data as abnormal entity data; Determine respectively whether the process central processing unit memory and process storage memory corresponding to the process entity type meet the process service level target index, if so, determine the target current abnormal index data as normal entity data; if not, determine the target current abnormal index data as abnormal entity data; The abnormal entity is determined according to the entity types respectively corresponding to the abnormal entity data.

8. A data anomaly detection and analysis device, characterized in that: include: A data attribute description information acquisition module is used to acquire a data set to be analyzed for anomaly detection, and to acquire data attribute description information corresponding to a plurality of data to be analyzed for anomaly detection in the data set to be analyzed for anomaly detection; A current abnormal indicator data determination module, used to determine at least one current abnormal indicator data according to the data attribute description information by a preset abnormal data detection method; The abnormal entity and abnormal entity data determination module is used to obtain the machine entity type and process entity type corresponding to each current abnormal indicator data, and determine the abnormal entity and abnormal entity data through the preset service level target indicator; The output feedback module is used to output and feedback the current abnormal indicator data, abnormal entities and abnormal entity data to the user.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data anomaly detection and analysis method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data anomaly detection and analysis method according to any one of claims 1 to 7 when executed.