Data mart anomaly detection method and device, computer device and storage medium

Abstract

The embodiment of the application belongs to the field of base frame operation and maintenance, and relates to a data mart abnormality detection method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining task blood relationship information of a data mart; for each application in the data mart, determining a fluctuation detection link of the application according to the task blood relationship information; determining the task weight and the repetition degree of each task in the link according to the task blood relationship information; obtaining the task log of the task to extract the running index of the task; calculating the task fluctuation value of the task according to the task weight, the repetition degree and the running index; calculating the application fluctuation value of the application based on the task fluctuation value, and calculating the mart fluctuation value of the data mart according to the application fluctuation value; comparing the mart fluctuation value with a standard mart fluctuation value to obtain a data mart abnormality detection result. In addition, the application also relates to blockchain technology, and the task blood relationship information can be stored in the blockchain. The application can discover abnormalities in the data mart in advance and give a warning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of base frame operation and maintenance and big data, and particularly relates to a data mart anomaly detection method and device, computer equipment and a storage medium. BACKGROUND

[0002] A data mart is a data cube oriented to decision analysis requirements, which is stored and generated in a multi-dimensional manner to meet the needs of a specific department or user. The data mart can include multiple applications, and each application can be composed of multiple tasks. Fluctuations in the repetition, timeliness and cost of tasks can affect the application or data mart, and even cause production accidents. Therefore, anomaly detection of the data mart is particularly important.

[0003] Current data mart anomaly detection techniques usually perform top-down troubleshooting after the mart has an anomaly. However, at this time, the anomaly has already occurred, and the data mart and the applications therein have been affected, and it is impossible to discover possible anomalies in advance and provide early warnings. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a data mart anomaly detection method, device, computer equipment and storage medium to discover possible anomalies in the data mart in advance.

[0005] To solve the above technical problems, the embodiments of the present application provide a data mart anomaly detection method, which adopts the following technical solutions:

[0006] Obtain task blood relationship information of a data mart;

[0007] For each application in the data mart, determine a fluctuation detection link of the application according to the task blood relationship information;

[0008] For each task in the fluctuation detection link, determine a task weight and a repetition degree of the task according to the task blood relationship information;

[0009] Obtain a task log of the task to extract a running index of the task from the task log;

[0010] Calculate a task fluctuation value of the task according to the task weight, the repetition degree and the running index;

[0011] Calculate an application fluctuation value of the application based on the task fluctuation value, and calculate a mart fluctuation value of the data mart according to the application fluctuation value;

[0012] Compare the mart fluctuation value with a preset standard mart fluctuation value to obtain a data mart anomaly detection result.

[0013] To solve the above technical problems, the embodiment of the application further provides a data mart anomaly detection device, which adopts the technical scheme as follows:

[0014] The blood relationship obtaining module is configured to obtain task blood relationship information of the data mart;

[0015] The link determining module is configured to determine, for each application in the data mart, a fluctuation detection link of the application according to the task blood relationship information;

[0016] The task determining module is configured to determine, for each task in the fluctuation detection link, a task weight and a repetition degree of the task according to the task blood relationship information;

[0017] The index extracting module is configured to obtain a task log of the task to extract a running index of the task from the task log;

[0018] The task calculating module is configured to calculate a task fluctuation value of the task according to the task weight, the repetition degree and the running index;

[0019] The mart calculating module is configured to calculate an application fluctuation value of the application based on the task fluctuation value and calculate a mart fluctuation value of the data mart according to the application fluctuation value;

[0020] The result generating module is configured to compare the mart fluctuation value with a preset standard mart fluctuation value to obtain a data mart anomaly detection result.

[0021] To solve the above technical problems, the embodiment of the application further provides a computer device, which adopts the technical scheme as follows:

[0022] Obtain task blood relationship information of a data mart;

[0023] For each application in the data mart, determine a fluctuation detection link of the application according to the task blood relationship information;

[0024] For each task in the fluctuation detection link, determine a task weight and a repetition degree of the task according to the task blood relationship information;

[0025] Obtain a task log of the task to extract a running index of the task from the task log;

[0026] Calculate a task fluctuation value of the task according to the task weight, the repetition degree and the running index;

[0027] calculate an application fluctuation value of the application based on the task fluctuation value, and calculate a market fluctuation value of the data mart based on the application fluctuation value;

[0028] compare the market fluctuation value with a preset standard market fluctuation value to obtain a data mart anomaly detection result.

[0029] To solve the above technical problems, the embodiment of the application further provides a computer readable storage medium, which adopts the technical scheme as follows:

[0030] obtain task blood relationship information of a data mart;

[0031] For each application in the data mart, determine a fluctuation detection link of the application according to the task blood relationship information;

[0032] For each task in the fluctuation detection link, determine a task weight and a repetition degree of the task according to the task blood relationship information;

[0033] obtain a task log of the task to extract a running index of the task from the task log;

[0034] calculate a task fluctuation value of the task according to the task weight, the repetition degree and the running index;

[0035] calculate an application fluctuation value of the application based on the task fluctuation value, and calculate a market fluctuation value of the data mart based on the application fluctuation value;

[0036] compare the market fluctuation value with a preset standard market fluctuation value to obtain a data mart anomaly detection result.

[0037] Compared with the prior art, the embodiment of the application has the following beneficial effects: the task blood relationship information is obtained, which records the upstream and downstream correlation between tasks and between tasks and applications in the data mart, and the fluctuation detection link of the application can be determined based on the task blood relationship information; for each task in the fluctuation detection link, the task weight and the repetition degree of the task are determined according to the task blood relationship information, wherein the task weight measures the support degree of the task to the downstream application, and can reflect the importance of the task, and the repetition degree measures the position of the task in the link; then the running index reflecting the running condition of the task is extracted according to the task log; the task fluctuation value of the task can be calculated according to the task weight, the repetition degree and the running index, the application fluctuation value of the application can be calculated based on the task fluctuation value, the market fluctuation value of the data mart can be calculated according to the application fluctuation value, the data mart anomaly detection result is obtained by comparing the market fluctuation value with a preset standard market fluctuation value; the automatic calculation and detection of fluctuation are realized, the fluctuation anomaly can be discovered in advance before the anomaly amplification, and the generation of the data mart anomaly is reduced. Attached Figure Description

[0038] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0040] Figure 2 This is a flowchart of an embodiment of the data mart anomaly detection method according to this application;

[0041] Figure 3 This is a schematic diagram of a data mart according to this application;

[0042] Figure 4 This is a schematic diagram of the structure of one embodiment of the data mart anomaly detection device according to this application;

[0043] Figure 5 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0045] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0046] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0047] likeFigure 1 As shown, the system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0048] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0049] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0050] The server 105 can be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.

[0051] It should be noted that the data mart anomaly detection method provided by the embodiments of the present application is generally executed by a server, and accordingly, the data mart anomaly detection apparatus is generally provided in a server.

[0052] It should be understood that Figure 1 The number of terminal devices, networks, and servers in

[0053] With reference to Figure 2 , a flowchart of one embodiment of the data mart anomaly detection method according to the present application is shown. The data mart anomaly detection method includes the following steps:

[0054] In step S201, task blood relationship information of a data mart is acquired.

[0055] In the present embodiment, the electronic device (for example, the server 105) on which the data mart anomaly detection method runs can acquire the task blood relationship information of the data mart in the following manners. Figure 1The server (as shown) can communicate with the terminal through wired or wireless connection. It should be noted that the wireless connection can include, but is not limited to, 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection.

[0056] Specifically, first, the task blood relationship information of the data mart is acquired. The data mart includes multiple applications, and each application can be composed of multiple tasks. The tasks have upstream and downstream relationships or dependency relationships, and the association relationship is similar to blood relationship. For example, the processing result of task A is input to task B, task B depends on task A, task A is the upstream task of task B, and task B is the downstream task of task A. The tasks and the applications also have upstream and downstream relationships or dependency relationships. For example, the processing result of task B is input to application C, application C depends on task B, and application C is downstream of task B. The task blood relationship information records the association relationship between the tasks in the data mart, and can also record the association relationship between the tasks and the applications.

[0057] Figure 3 In an embodiment, the schematic diagram of the data mart is shown in FIG. 1. Specifically, referring to FIG. 1, Figure 3 There are application A and application B in the data mart, and there are task 1, task 2, task 3, task 4, task 5, task 6, and task 7. The association relationship between the tasks and the applications is shown in FIG. 1. Figure 3 The task blood relationship information can also be acquired.

[0058] In step S202, for each application in the data mart, the task blood relationship information is used to determine the fluctuation detection link of the application.

[0059] Specifically, each application in the data mart has at least one task link. The task link refers to the path from the task to the application, and the tasks and the applications in the path have blood relationship, so the task link of the application can be determined through the task blood relationship information. For example, in Figure 3 , task 1-task 4-task 6-application A is one task link of application A; task 2-task 5-task 6-application A is another task link of application A; and task 3-task 7-application B is the only task link of application B.

[0060] Through the preset link determination strategy, the fluctuation detection link can be determined from the task link of the application, and the fluctuation detection and early warning of the application and the data mart are performed based on the fluctuation detection link.

[0061] In step S203, for each task in the fluctuation detection link, the task blood relationship information is used to determine the task weight and the repetition degree of the task.

[0062] Specifically, the application includes at least one task in the fluctuation detection link. For each task, it is necessary to determine the task weight of the task and the repetition degree. The task weight is used to measure the support degree of the task to the downstream application. The more applications the task supports, that is, the more applications the task can reach, the greater the task weight of the task.

[0063] The repetition degree is related to the position of the task in the fluctuation detection link. The support degree of the task to the downstream application and the position of the task in the fluctuation detection link can be obtained through the task blood relationship. Therefore, the task weight and the repetition degree of the task can be determined through the task blood relationship information.

[0064] Step S204, obtaining the task log of the task to extract the running index of the task from the task log.

[0065] Specifically, the task log of the task is obtained, and the task log records the running of the task, so the running index of the task can be extracted from the task log.

[0066] The running index can be an index recording the running state of the task. In an embodiment, the running index includes running time efficiency and running cost. The running time efficiency represents the time calculated by the task or the application, which is the running time of the task or the application, and is used to measure the running time of the task or the application. Figure 3 Taking task 1 in the application as an example, the running time efficiency of task 1 obtained from the task log is 1 hour, indicating that task 1 runs for 1 hour, and the running cost of task 1 is 10 core hours, wherein 1 core hour represents that a single CPU (central processing unit) runs for 1 hour, and the core hour is used to measure the CPU resources occupied by the task.

[0067] Step S205, calculating the task fluctuation value of the task according to the task weight, the repetition degree and the running index.

[0068] Specifically, the repetition degree and each running index have a base value and a current value. The current value can be the current value, and the base value can be the value at a certain time point in the past. According to the base value and the current value of the repetition degree and each running index, the fluctuation value of the repetition degree and each running index can be calculated first.

[0069] Then, the task fluctuation value of the task is calculated according to the fluctuation value of the repetition degree and the fluctuation value of each running index. The fluctuation value of the repetition degree and the fluctuation value of each running index can be calculated first, and then the task weight is calculated to calculate the task fluctuation value. The task fluctuation value is used to measure the fluctuation of the task.

[0070] Step S206, calculating the application fluctuation value of the application based on the task fluctuation value, and calculating the market fluctuation value of the data mart according to the application fluctuation value.

[0071] Specifically, each task in the fluctuation detection link calculates a task fluctuation value, and then based on the task fluctuation values of the tasks in the fluctuation detection link, an application fluctuation value of the application can be calculated. The data mart has multiple applications, and based on the application fluctuation value of each application, a mart fluctuation value of the entire data mart can be calculated.

[0072] The task fluctuation value is used to measure the fluctuation of the task, the application fluctuation value is used to measure the fluctuation of the application, and the mart fluctuation value is used to measure the fluctuation of the mart; the greater the fluctuation value, the greater the fluctuation.

[0073] In step S207, the mart fluctuation value is compared with a preset standard mart fluctuation value to obtain a data mart anomaly detection result.

[0074] Specifically, the standard mart fluctuation value can be set in advance, and there are multiple standard mart fluctuation values, which form multiple evaluation intervals. The evaluation interval divides the fluctuation into several states, for example, the states corresponding to the evaluation interval can include slight, normal, larger fluctuation, and abnormal fluctuation. The mart fluctuation value is compared with each standard mart fluctuation value to determine the evaluation interval in which the mart fluctuation value is located, and the state corresponding to the evaluation interval and the mart fluctuation value are taken as the data mart anomaly detection result; when the mart fluctuation value is in the slight or normal state, it indicates that the data mart does not have an anomaly, otherwise, it indicates that the data mart has an anomaly. When it is detected that the data mart has an anomaly, an alarm can be given in time so as to discover the problem in the data mart in time and make corrections, thereby avoiding the amplification of the anomaly to a production problem and affecting the normal operation of the data mart.

[0075] In this embodiment, the task blood relationship information is obtained, which records the upstream and downstream association between tasks and between tasks and applications in the data mart. Based on the task blood relationship information, the fluctuation detection link of the application can be determined. For each task in the fluctuation detection link, the task weight and the repetition degree of the task are determined according to the task blood relationship information, wherein the task weight measures the support degree of the task to the downstream application and can reflect the importance of the task, and the repetition degree measures the position of the task in the link. Then, the running index reflecting the running status of the task is extracted according to the task log. According to the task weight, the repetition degree and the running index, the task fluctuation value of the task can be calculated, based on the task fluctuation value, the application fluctuation value of the application can be calculated, and based on the application fluctuation value, the mart fluctuation value of the data mart can be calculated. The mart fluctuation value is compared with a preset standard mart fluctuation value to obtain a data mart anomaly detection result. The application realizes automatic calculation and detection of fluctuation, can discover fluctuation anomaly in advance before the anomaly is amplified, and thereby reduces the generation of data mart anomaly.

[0076] Further, the step S202 can comprise: determining each task link of the application according to the task blood relationship information; counting the link length of each task link; and selecting the task link with the maximum link length as the fluctuation detection link of the application.

[0077] Specifically, each application in the data mart has at least one task link. The task link refers to the path from the task to the application, and the tasks in the path have blood relationship with the application. The task link of the application can be determined through the task blood relationship information.

[0078] The task link has a link length, which refers to the number of tasks in the task link. For example, in Figure 3 , the task 1-task 4-task 6-application A is a task link of the application A, which contains 3 tasks, and the link length is 3. The link length of each task link is counted, and the task link with the maximum link length is selected as the fluctuation detection link of the application.

[0079] Further, the step of selecting the task link with the maximum link length as the fluctuation detection link of the application can comprise: when the task link with the maximum link length is one, determining the task link as the fluctuation detection link; and when the task link with the maximum link length is more than one, randomly selecting a task link from the task links with the maximum link length as the fluctuation detection link of the application.

[0080] Specifically, after counting the link length of each task link, the task link with the maximum link length is selected. If the task link with the maximum link length is only one, the task link is directly determined as the fluctuation detection link. For example, if the application has three task links with link lengths of 4, 5, and 6, the task link with the link length of 6 is selected as the fluctuation detection link.

[0081] If the task link with the maximum link length is more than one, for example, in Figure 3 , the application A has “task 1-task 4-task 6-application A”, “task 2-task 4-task 6-application A”, “task 2-task 5-task 6-application A”, and “task 3-task 5-task 6-application A”, a total of 4 task links, and the link lengths are all 3, i.e. the task links with the maximum link length are 4. Then, a task link is randomly selected from the task links with the maximum link length as the fluctuation detection link.

[0082] In this embodiment, when there is only one task link with the maximum link length, the task link is directly determined as the fluctuation detection link; when there is more than one task link with the maximum link length, the fluctuation detection link is randomly determined from the task links with the maximum link length, thereby ensuring that a fluctuation detection link can be obtained.

[0083] In this embodiment, after obtaining the application's task links based on the bloodline information, the link length of each task link is counted, and thus the fluctuation detection link is determined from multiple task links based on the link length.

[0084] Furthermore, step S203 may include: calculating the number of applications associated with each task in the statistical data marketplace based on the task lineage information; calculating the task weight of each task in the fluctuation detection link based on the number of applications; and determining the task repetition based on the task's level in the fluctuation detection link.

[0085] Specifically, for each task in the data mart, based on task lineage information, it's possible to count how many applications the task ultimately reaches, thus obtaining the number of applications associated with the task. Figure 3 For example, after Task 3, you can eventually reach two applications, Application A and Application B. Therefore, the number of applications associated with Task 3 is 2. For Task 1, Task 2, Task 4, Task 5, Task 6 and Task 7, the number of applications associated with them is 1.

[0086] The number of applications associated with a task refers to the number of applications supported by the task. This number is used to calculate the task weight, which is calculated from the perspective of the entire data mart. In one embodiment, the task weight can be expressed as:

[0087]

[0088] Among them, w i N represents the task weight. i This represents the number of applications associated with a task, and n represents the number of tasks in the data mart. (Combined) Figure 3 According to formula (1), it is easy to see that the task weight of task 3 is 2 / 8, and the task weights of tasks 1, 2, 4, 5, 6 and 7 are 1 / 8.

[0089] The fluctuation detection link has at least one task, and the task repeatability can be determined based on its hierarchy within the link. Figure 3 In the example, assuming the fluctuation detection link of application A is task 1-task 4-task 6-application A, task 1 is in the first level, so its repeatability value is 1; task 4 is in the second level, so its repeatability value is 2; and task 6 is in the third level, so its repeatability value is 3.

[0090] In this embodiment, the application quantity of the application associated with the task is counted from the overall perspective of the data mart, so as to calculate the task weight of the task; according to the level of the task in the fluctuation detection link, the repetition degree of the task can be determined, so as to ensure the calculation of the task fluctuation value.

[0091] Further, the repetition degree includes a base period repetition degree and a current period repetition degree, and the operation index includes a base period operation index and a current period operation index. The step S205 can include: calculating a repetition degree fluctuation value according to the base period repetition degree and the current period repetition degree, and calculating an operation index fluctuation value according to the base period operation index and the current period operation index; determining an index weight of the repetition degree fluctuation value and the operation index fluctuation value based on a preset weight algorithm; and calculating the task weight, the repetition degree fluctuation value and the operation index fluctuation value with the index weight, to obtain the task fluctuation value of the task.

[0092] Specifically, when the repetition degree and the operation index are obtained, the repetition degree and the operation index at two time points are actually obtained. The repetition degree includes a base period repetition degree and a current period repetition degree. The base period repetition degree can be a repetition degree at a past time point or a repetition degree at a preset first time point. The current period repetition degree can be a repetition degree at a current time point or a repetition degree at a preset second time point. The second time point is later than the first time point.

[0093] Similarly, the operation index includes a base period operation index and a current period operation index. The base period operation index can be an operation index at a past time point or an operation index at a preset first time point. The current period operation index can be an operation index at a current time point or an operation index at a preset second time point. The second time point is later than the first time point.

[0094] The repetition degree fluctuation value can be calculated according to the base period repetition degree and the current period repetition degree. In an embodiment, the repetition degree fluctuation value can be represented as follows:

[0095]

[0096] wherein, ΔR i represents the repetition degree fluctuation value, represents the current period repetition degree, represents the base period repetition degree.

[0097] Similarly, the operation index fluctuation value can be calculated according to the base period operation index and the current period operation index. In an embodiment, the operation index includes a time efficiency S i and an operation cost C i The operation index fluctuation value includes a time efficiency fluctuation value and a cost fluctuation value. In an embodiment, the time efficiency fluctuation value can be represented as follows:

[0098]

[0099] wherein, ΔS i % represents the time efficiency fluctuation value, represents the current operation time efficiency, represents the base operation time efficiency.

[0100] The cost fluctuation value can be represented as follows:

[0101]

[0102] wherein, ΔC i % represents the cost fluctuation value, represents the current operation cost, represents the base operation cost.

[0103] In the present application, the redundancy or redundancy fluctuation value, the operation index or operation index fluctuation value has an index weight, so as to distinguish the importance thereof in the calculation of the task fluctuation value. The calculation of the index weight can be realized by a preset weight algorithm, for example, can be calculated by CRITIC algorithm, analytic hierarchy process, Relief algorithm, etc.

[0104] In one embodiment, the weight algorithm determines the index weight according to the priority of the application. The priority of the application is obtained, and the preset index weight is queried according to the priority. For example, the high-priority application pays more attention to the operation time efficiency, and can weaken the operation cost and the redundancy, so that the weight ratio of the operation time efficiency, the operation cost and the redundancy of the task supporting the high-priority application is 6:2:2; for the task of the medium-priority application, the weight ratio of the operation time efficiency, the operation cost and the redundancy is 4:3:3; for the task of the low-priority application, the weight ratio of the operation time efficiency, the operation cost and the redundancy is 1:6:3.

[0105] Then, the task fluctuation value of the task is calculated based on the task weight and the redundancy fluctuation value and the operation index fluctuation value (time efficiency fluctuation value and cost fluctuation value) with the index weight. In one embodiment, the task fluctuation value is represented as follows:

[0106]

[0107] wherein, C Ti is the task fluctuation value; w i is the task weight; ΔR i % is the redundancy fluctuation value, is the index weight of the redundancy fluctuation value; ΔS i % is the time efficiency fluctuation value, is the index weight of the time efficiency fluctuation value; ΔC i % is the cost fluctuation value, The index weight of the cost fluctuation value.

[0108] In this embodiment, the repetition fluctuation value is calculated according to the base repetition degree and the current repetition degree, and the operation index fluctuation value is calculated according to the base operation index and the current operation index; the index weight of the repetition fluctuation value and the operation index fluctuation value is determined based on the weight algorithm, so as to distinguish the importance of the repetition fluctuation value and the operation index fluctuation value; the task fluctuation value is calculated according to the task weight and the repetition fluctuation value and the operation index fluctuation value with the index weight, so as to measure the task fluctuation.

[0109] Further, the step S206 can include: performing linear operation on the task fluctuation values of the tasks in the fluctuation detection link to obtain the application fluctuation value of the application; and performing linear operation on the application fluctuation values of the applications in the data mart to obtain the mart fluctuation value.

[0110] Specifically, after obtaining the task fluctuation values of the tasks in the fluctuation detection link, the linear operation is performed on the task fluctuation values of the tasks, so as to obtain the application fluctuation value of the application. In one embodiment, the task fluctuation values of the tasks in the fluctuation detection link are accumulated, so as to obtain the application fluctuation value of the application.

[0111] After obtaining the application fluctuation values of the applications in the data mart, the linear operation is performed on the application fluctuation values of the applications, so as to obtain the mart fluctuation value of the data mart. In one embodiment, the application fluctuation values of the applications in the data mart are accumulated, so as to obtain the mart fluctuation value of the data mart.

[0112] In this embodiment, the linear operation is performed on the task fluctuation values of the tasks in the fluctuation detection link, so as to obtain the application fluctuation value, and the linear operation is performed on the application fluctuation values of the applications in the data mart, so as to obtain the mart fluctuation value, thereby realizing the measurement of the application fluctuation and the mart fluctuation.

[0113] Further, after the step S207, the method can further include: when it is determined according to the data mart abnormality detection result that the data mart is in the fluctuation abnormality state, drilling down the mart fluctuation value to obtain the abnormality diagnosis information.

[0114] Specifically, when it is determined according to the data mart abnormality detection result that the data mart is in the fluctuation abnormality state, the drilling down is performed on the mart fluctuation value, which can specifically include: obtaining the application fluctuation value of each application, comparing the application fluctuation value with a preset standard application fluctuation value, if the application fluctuation value is greater than the standard application fluctuation value, determining that the application is in the abnormality state; and obtaining the task fluctuation value associated with the application, comparing the task fluctuation value with a preset standard task fluctuation value, if the task fluctuation value is greater than the preset standard task fluctuation value, determining that the task is in the abnormality state.

[0115] According to the above drilling strategy, the abnormal diagnosis information is generated according to the detected abnormal application and abnormal task, so that the relevant staff can adjust and repair the application or task according to the abnormal diagnosis information.

[0116] It can be understood that the detected abnormality in the present application refers to an abnormality compared with the normal level, but does not necessarily affect the operation of the data mart, thereby realizing the pre-discovery and alarm of the abnormality.

[0117] In the embodiment, when it is determined according to the data mart abnormality detection result that the data mart is in a fluctuation abnormal state, the market fluctuation value is drilled to analyze the abnormal application or task layer by layer and generate abnormal diagnosis information, so that the abnormality can be eliminated in time.

[0118] It should be emphasized that, in order to further ensure the privacy and security of the task blood relationship information and the task log, the task blood relationship information and the task log can also be stored in a node of a block chain.

[0119] The block chain referred to in the present application is a new application mode of distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and other computer technologies. The block chain (block chain) is essentially a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block contains a batch of network transaction information, used to verify the validity (anti-fake) of the information and generate the next block. The block chain can include a block chain underlying platform, a platform product service layer and an application service layer.

[0120] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. For example, the task log is analyzed and the required running index is extracted by artificial intelligence. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0121] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. Several major directions.

[0122] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through computer readable instructions, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiment methods. Among them, the storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0123] It should be understood that although each step in the flowchart of the accompanying drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0124] Further referring to Figure 2 , as an implementation of the method shown in Figure 2 , the present application provides an embodiment of a data mart anomaly detection device, which corresponds to the method embodiment shown in Figure 4 , and the device can be specifically applied to various electronic devices.

[0125] As shown in Figure 5 , the data mart anomaly detection device 300 described in the embodiment includes a blood relationship acquisition module 301, a link determination module 302, a task determination module 303, an index extraction module 304, a task calculation module 305, a mart calculation module 306, and a result generation module 307, wherein:

[0126] The blood relationship acquisition module 301 is configured to acquire task blood relationship information of a data mart.

[0127] The link determination module 302 is configured to determine a fluctuation detection link of each application in the data mart according to the task blood relationship information.

[0128] The task determination module 303 is configured to determine a task weight and a repetition degree of each task in the fluctuation detection link according to the task blood relationship information.

[0129] The index extraction module 304 is configured to acquire a task log of the task, and extract a running index of the task from the task log.

[0130] The task calculation module 305 is configured to calculate a task fluctuation value of the task according to the task weight, the repetition degree and the running index.

[0131] The market calculation module 306 is configured to calculate an application fluctuation value of the application based on the task fluctuation value, and calculate a market fluctuation value of the data market according to the application fluctuation value.

[0132] The result generation module 307 is configured to compare the market fluctuation value with a preset standard market fluctuation value, and obtain a data market anomaly detection result.

[0133] In the embodiment, the task blood relationship information is acquired, which records the upstream and downstream association relationship between tasks and between the task and the application in the data market. The fluctuation detection link of the application can be determined based on the task blood relationship information. For each task in the fluctuation detection link, the task weight and the repetition degree of the task are determined according to the task blood relationship information, wherein the task weight measures the support degree of the task to the downstream application, and can reflect the importance of the task, and the repetition degree measures the position of the task in the link. Then, the running index reflecting the running condition of the task is extracted according to the task log. The task fluctuation value of the task can be calculated according to the task weight, the repetition degree and the running index. The application fluctuation value of the application can be calculated based on the task fluctuation value. The market fluctuation value of the data market can be calculated according to the application fluctuation value. The data market anomaly detection result is obtained by comparing the market fluctuation value with a preset standard market fluctuation value. The automatic calculation and detection of fluctuation are realized, the fluctuation anomaly can be discovered in advance before the anomaly amplification, and thus the generation of the data market anomaly is reduced.

[0134] In some optional implementation manners of the embodiment, the link determination module 302 can include a link determination sub-module, a length statistics sub-module and a fluctuation determination sub-module, wherein:

[0135] The link determination sub-module is configured to determine, for each application in the data market, a task link of the application according to the task blood relationship information.

[0136] The length statistics sub-module is configured to count the link length of each task link.

[0137] The fluctuation determination sub-module is configured to select the task link with the maximum link length as the fluctuation detection link of the application.

[0138] In the embodiment, after obtaining each task link of the application according to the blood relationship information, the link length of each task link is counted, so that the fluctuation detection link is determined from the multiple task links according to the link length.

[0139] In some optional implementations of the embodiment, the fluctuation determination submodule can include a link determination unit and a random selection unit, wherein:

[0140] The link determination unit is configured to determine the task link with the maximum link length as the fluctuation detection link when the task link with the maximum link length is one.

[0141] The random selection unit is configured to randomly select a task link from the task links with the maximum link length as the fluctuation detection link to be applied when the task links with the maximum link length are more than one.

[0142] In the embodiment, when the task link with the maximum link length is one, the task link is directly determined as the fluctuation detection link; when the task links with the maximum link length are more than one, the fluctuation detection link is determined in a random manner from the task links with the maximum link length, so that the fluctuation detection link can be obtained.

[0143] In some optional implementations of the embodiment, the task determination module 303 can include a quantity determination submodule, a weight calculation submodule, and a repetition degree determination submodule, wherein:

[0144] The quantity determination submodule is configured to count the number of applications associated with each task in the data mart according to the task blood relationship information.

[0145] The weight calculation submodule is configured to calculate the task weight of each task in the fluctuation detection link according to the number of applications.

[0146] The repetition degree determination submodule is configured to determine the repetition degree of a task according to the level of the task in the fluctuation detection link.

[0147] In the embodiment, the number of applications associated with each task is counted from the overall perspective of the data mart, so that the task weight of the task is calculated; the repetition degree of the task is determined according to the level of the task in the fluctuation detection link, so that the calculation of the subsequent task fluctuation value is ensured.

[0148] In some optional implementations of the embodiment, the repetition degree includes a base period repetition degree and a current period repetition degree, and the operation index includes a base period operation index and a current period operation index; the task calculation module can include a fluctuation calculation submodule, a weight determination submodule, and a task calculation submodule, wherein:

[0149] The fluctuation calculation submodule is configured to calculate a repetition degree fluctuation value according to the base period repetition degree and the current period repetition degree, and calculate an operation index fluctuation value according to the base period operation index and the current period operation index.

[0150] The weight determination sub-module is configured to determine the index weight of the redundancy fluctuation value and the operation index fluctuation value based on a preset weight algorithm.

[0151] The task calculation sub-module is configured to calculate the task fluctuation value of the task based on the task weight and the redundancy fluctuation value and the operation index fluctuation value with the index weight.

[0152] In the embodiment, the redundancy fluctuation value is calculated based on the base period redundancy and the current period redundancy, and the operation index fluctuation value is calculated based on the base period operation index and the current period operation index; the index weight of the redundancy fluctuation value and the operation index fluctuation value is determined based on the weight algorithm, so as to distinguish the importance of the redundancy fluctuation value and the operation index fluctuation value; the task fluctuation value of the task is calculated based on the task weight and the redundancy fluctuation value and the operation index fluctuation value with the index weight, so as to measure the task fluctuation.

[0153] In some optional implementation of the embodiment, the market calculation module 306 can include an application calculation sub-module and a market calculation sub-module, wherein:

[0154] The application calculation sub-module is configured to perform linear operation on the task fluctuation values of the tasks in the fluctuation detection link to obtain the application fluctuation value of the application.

[0155] The market calculation sub-module is configured to perform linear operation on the application fluctuation values of the applications in the data market to obtain the market fluctuation value.

[0156] In the embodiment, the application fluctuation value can be obtained by performing linear operation on the task fluctuation values of the tasks in the fluctuation detection link, and the market fluctuation value can be obtained by performing linear operation on the application fluctuation values of the applications in the data market, so as to measure the application fluctuation and the market fluctuation.

[0157] In some optional implementation of the embodiment, the data market anomaly detection apparatus 300 can further include a drilling module configured to drill down the market fluctuation value to obtain abnormal diagnosis information when it is determined according to the data market anomaly detection result that the data market is in a fluctuation abnormal state.

[0158] In the embodiment, when it is determined according to the data market anomaly detection result that the data market is in a fluctuation abnormal state, the market fluctuation value is drilled down to analyze the abnormal application or task layer by layer and generate abnormal diagnosis information, so as to eliminate the abnormality in time.

[0159] To solve the above technical problem, the embodiment of the present application further provides a computer device. For details, please refer to Figure 5 , ​ The basic structure block diagram of the computer device of the embodiment is shown in FIG. 4.

[0160] The computer device 4 includes a memory 41, a processor 42, and a network interface 43, which are communicatively connected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure, but it should be understood that not all of the shown components are required to be implemented, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0161] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, and the like.

[0162] The memory 41 includes at least one type of readable storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Of course, the memory 41 can also include both the internal storage unit and the external storage device of the computer device 4. In the present embodiment, the memory 41 is generally used to store an operating system and various application software installed in the computer device 4, such as computer readable instructions of the data mart anomaly detection method, and the like. In addition, the memory 41 can also be used to temporarily store various data that have been output or will be output.

[0163] The processor 42 may, in some embodiments, be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In the present embodiment, the processor 42 is configured to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions of the data mart anomaly detection method.

[0164] The network interface 43 may include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0165] The computer device provided in the present embodiment can execute the data mart anomaly detection method described above. The data mart anomaly detection method herein can be the data mart anomaly detection method of any of the embodiments described above.

[0166] In the present embodiment, the task blood relationship information is obtained, which records the upstream and downstream association between tasks and between tasks and applications in the data mart. Based on the task blood relationship information, the fluctuation detection link of the application can be determined. For each task in the fluctuation detection link, the task weight and the repetition degree of the task are determined according to the task blood relationship information, wherein the task weight measures the support degree of the task to the downstream application, and can reflect the importance of the task, and the repetition degree measures the position of the task in the link. Then, the running index reflecting the running status of the task is extracted according to the task log. According to the task weight, the repetition degree and the running index, the task fluctuation value of the task can be calculated, and based on the task fluctuation value, the application fluctuation value of the application can be calculated. According to the application fluctuation value, the mart fluctuation value of the data mart can be calculated. The mart fluctuation value is compared with the preset standard mart fluctuation value to obtain the data mart anomaly detection result. The present application realizes automatic calculation and detection of fluctuation, and can discover fluctuation anomaly in advance before the anomaly amplification, thereby reducing the generation of data mart anomaly.

[0167] The present application also provides another implementation, i.e., a computer readable storage medium storing computer readable instructions, which can be executed by at least one processor to enable the at least one processor to perform the steps of the data mart anomaly detection method as described above.

[0168] In the embodiment, the task blood relationship information is acquired, which records the upstream and downstream association between tasks and between tasks and applications in the data mart, and based on the task blood relationship information, the fluctuation detection link of the application can be determined; for each task in the fluctuation detection link, the task weight and the repetition degree of the task are determined according to the task blood relationship information, wherein the task weight measures the support degree of the task to the downstream application, and can reflect the importance of the task, and the repetition degree measures the position of the task in the link; then the running index reflecting the running status of the task is extracted according to the task log; according to the task weight, the repetition degree and the running index, the task fluctuation value of the task can be calculated, based on the task fluctuation value, the application fluctuation value of the application can be calculated, according to the application fluctuation value, the market fluctuation value of the data mart can be calculated, the market fluctuation value is compared with the preset standard market fluctuation value, and the data mart anomaly detection result is obtained; the automatic calculation and detection of the fluctuation are realized, the fluctuation anomaly can be discovered in advance before the anomaly amplification, and thus the generation of the data mart anomaly is reduced.

[0169] Through the description of the above implementation manner, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present application.

[0170] Obviously, the above-described embodiments are only some embodiments of the present application, not all embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be realized in many different forms, and on the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent replacements to some technical features. Any equivalent structure made by using the contents of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the patent protection scope of the present application.

Claims

1. A data mart anomaly detection method, characterized in that, Includes the following steps: Obtain task lineage information from the data mart; For each application in the data mart, the fluctuation detection link of the application is determined based on the task lineage information; For each task in the fluctuation detection chain, the task weight and repetition degree of the task are determined based on the task lineage information. Obtain the task log of the task, and extract the task's operation metrics from the task log; The task fluctuation value is calculated based on the task weight, the repetition rate, and the operational metrics. The application fluctuation value of the application is calculated based on the task fluctuation value, and the data mart fluctuation value is calculated based on the application fluctuation value, wherein the data mart contains several applications; The market fluctuation value is compared with the preset standard market fluctuation value to obtain the data market anomaly detection result; The step of calculating the task fluctuation value based on the task weight, the repetition rate, and the performance indicators specifically includes the following steps: Based on the repeatability and the base and current values ​​of each operating indicator, the fluctuation values ​​of the repeatability and each operating indicator can be calculated first. Then, the task fluctuation value can be calculated based on the task weight, the fluctuation value of the repeatability and the fluctuation value of each operating indicator. The steps of calculating the application volatility value based on the task volatility value and calculating the data mart volatility value based on the application volatility value specifically include the following steps: Calculate the task fluctuation value for each task in the fluctuation detection link, and then calculate the application fluctuation value based on the task fluctuation value of each task in the fluctuation detection link. The overall data mart volatility can be calculated based on the application volatility of each application.

2. The data mart anomaly detection method according to claim 1, characterized in that, The step of determining the fluctuation detection link of each application in the data mart based on the task lineage information includes: For each application in the data mart, the task links of the application are determined based on the task lineage information; Calculate the link length of each task link; The task link with the longest link length is selected as the fluctuation detection link for the application.

3. The data mart anomaly detection method according to claim 2, characterized in that, The step of selecting the task link with the longest link length as the fluctuation detection link for the application includes: When there is only one task link with the maximum link length, the task link is identified as the fluctuation detection link. When there is more than one task link with the maximum link length, a task link with the maximum link length is randomly selected from each task link as the fluctuation detection link of the application.

4. The data mart anomaly detection method according to claim 1, characterized in that, The step of determining the task weight and repetition of each task in the fluctuation detection link based on the task lineage information includes: Based on the task lineage information, count the number of applications associated with each task in the data mart; Calculate the task weight of each task in the fluctuation detection link based on the number of applications; The repeatability of the task is determined based on its level in the fluctuation detection link.

5. The data mart anomaly detection method according to claim 1, characterized in that, The repeatability includes base period repeatability and current period repeatability, and the operating indicators include base period operating indicators and current period operating indicators. The step of calculating the task fluctuation value of the task based on the task weight, the repeatability, and the operating indicators includes: The repetition fluctuation value is calculated based on the base period repetition and the current period repetition, and the operating indicator fluctuation value is calculated based on the base period operating indicator and the current period operating indicator. Based on a preset weighting algorithm, the index weights of the repeatability fluctuation value and the operational index fluctuation value are determined. The task fluctuation value is obtained by calculating the task weight, the repetition fluctuation value with indicator weight, and the operation indicator fluctuation value.

6. The data mart anomaly detection method according to claim 1, characterized in that, The steps of calculating the application volatility value based on the task volatility value and calculating the data mart volatility value based on the application volatility value include: The application fluctuation value of the application is obtained by performing a linear operation on the task fluctuation values ​​of each task in the fluctuation detection link. The application volatility value of each application in the data mart is obtained by performing a linear operation on the application volatility value.

7. The data mart anomaly detection method according to claim 1, characterized in that, After the step of comparing the market fluctuation value with a preset standard market fluctuation value to obtain the data market anomaly detection result, the method further includes: When the data mart is determined to be in an abnormal fluctuation state based on the anomaly detection result, the fluctuation value of the data mart is drilled down to obtain anomaly diagnosis information.

8. A data mart anomaly detection device, characterized in that, include: The lineage acquisition module is used to obtain the task lineage relationship information from the data mart; The link determination module is used to determine the fluctuation detection link of each application in the data mart based on the task lineage information. The task determination module is used to determine the task weight and repetition of each task in the fluctuation detection link based on the task lineage information. The metric extraction module is used to obtain the task log of the task in order to extract the running metrics of the task from the task log; The task calculation module is used to calculate the task fluctuation value of the task based on the task weight, the repetition degree, and the operation indicators. The marketplace calculation module is used to calculate the application fluctuation value of the application based on the task fluctuation value, and to calculate the marketplace fluctuation value of the data marketplace based on the application fluctuation value, wherein the data marketplace contains several applications; The result generation module is used to compare the market fluctuation value with the preset standard market fluctuation value to obtain the data market anomaly detection result. The task calculation module is also used to calculate the fluctuation value of the repeatability and each operating indicator based on the base period value and current period value of the repeatability and each operating indicator, and then calculate the task fluctuation value of the task based on the task weight, the fluctuation value of the repeatability and the fluctuation value of each operating indicator. The marketplace calculation module is also used to calculate the task fluctuation value of each task in the fluctuation detection link, and then calculate the application fluctuation value of the application based on the task fluctuation value of each task in the fluctuation detection link. The marketplace calculation module is also used to calculate the marketplace fluctuation value of the entire data marketplace based on the application fluctuation value of each application.

9. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the data mart anomaly detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data mart anomaly detection method as described in any one of claims 1 to 7.

Images