Data anomaly positioning method and device based on time burying point sorting and pre-statistics

Abstract

The invention discloses a data anomaly positioning method and device based on time burying point sorting and pre-statistics, and the method comprises the steps: determining an index object set from all data indexes of target data before the data processing of the target data of a data warehouse; when data processing is carried out on the target data, time burying point processing is carried out on a data table set of the target data, the data processing time of each data table in the data table set is determined, and a first production list is obtained; performing index value statistics on the index object of each data table in the data table set to obtain a second production list; and according to index value statistical results of different data tables, determining an anomaly judgment result of the target data, and positioning to obtain a data processing node with data anomaly. According to the embodiment of the invention, the processing node with abnormal data can be efficiently positioned from the data processing chain of the data warehouse, so that the labor cost and the time cost of production operation are reduced, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for locating data anomalies based on time-based point sorting and pre-statistics. Background Art

[0002] With the development of big data technology, the scenarios of data warehouse design and related data processing derived from big data technology are becoming more and more abundant. Usually, in big data scenarios, data warehouse data processing from data producers to data consumers often requires a long processing chain. When consumers find data problems, they need to trace back from the point where the problem was found along the data processing chain to locate the point where the data problem was born. This process is not only time-consuming, but also delays the use of data and affects business operations. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to propose an efficient method and device for locating data anomalies based on time point sorting and pre-statistics.

[0004] To achieve the above purpose, an embodiment of the present invention provides a method for locating data anomalies based on time-based point sorting and pre-statistics, including the following steps: Before processing the target data of the data warehouse, determining an indicator object set from various data indicators of the target data; wherein the indicator object set includes at least one indicator object; When processing the target data, time-point processing is performed on the data table set of the target data to determine the data processing time of each data table in the data table set, and obtain a first production list; After performing index value statistics on the index objects of each data table in the data table set, performing compound sorting on the first production list to obtain a second production list; According to the statistical results of the indicator values ​​of different data tables in the second production list, the abnormality judgment result of the target data is determined and the data processing node where the data abnormality occurs is located.

[0005] In some embodiments, before processing the target data in the data warehouse, determining the indicator object set from various data indicators of the target data includes the following steps: Before data processing is performed on the target data of the data warehouse, the product quantity of the target data is determined as an indicator object in the indicator object set; Or, before processing the target data in the data warehouse, the product amount of the target data is determined as an indicator object in the indicator object set.

[0006] In some embodiments, when processing the target data, performing time embedding processing on a data table set of the target data, determining the data processing time of each data table in the data table set, and obtaining a first production list includes the following steps: When processing the target data, a source table, a target table and a process table of the target data are determined as a data table set of the target data; For the source table, the process table and the target table, a timestamp field is added; When writing data to the data table set of the target data, the system time of each data table is recorded in the timestamp field, the time embedding processing is completed, and the first production list is obtained; Wherein, the source table is used to represent the data source of the indicator object; The process table is used to characterize the processing process of the indicator data; The target table is used to represent the storage location of the indicator data.

[0007] In some embodiments, when writing data to the data table set of the target data, the system time of each data table is recorded in the timestamp field, the time embedding processing is completed, and the first production list is obtained, specifically: The system time recorded in the timestamp field of each of the data tables is sorted, and then each of the data tables is sorted according to the sorting result of the system time to obtain the first production list.

[0008] In some embodiments, after performing index value statistics on the index objects of each data table in the data table set, performing compound sorting on the first production list to obtain the second production list includes the following steps: According to the indicator type of the indicator object, counting statistics or summing statistics are performed on the indicator objects of each data table in the data table set to obtain an indicator value statistical result; A column is added to the first production list to record the statistical results of the indicator values, so as to obtain the second production list.

[0009] In some embodiments, determining the abnormality determination result of the target data and locating the data processing node where the data abnormality occurs according to the statistical results of the indicator values ​​of different data tables in the second production list includes the following steps: Determine a first index value of a source table and a second index value of a target table from the second production list; When the first index value is not equal to the second index value, it is determined that an abnormality exists in the data processing process of the second production list; When it is determined that an abnormality exists in the data processing process of the second production list, a process table where the data abnormality occurs is determined from the second production list according to the first indicator value and the second indicator value.

[0010] In some embodiments, when it is determined that there is an abnormality in the data processing process of the second production list, determining a process table where the data abnormality occurs from the second production list according to the first indicator value and the second indicator value includes the following steps: When it is determined that there is an abnormality in the data processing process of the second production list, the statistical results of the index values ​​of each data table in the second production list are compared one by one, and the process table where the data abnormality occurs is determined from the second production list: If the index value of the source table is less than the index value of the target table, a set of candidate process tables where data anomalies occur is determined from the second production list, and a candidate process table with the smallest index value is found from the set of candidate process tables to be determined as the process table where data anomalies occur; wherein the index value of each candidate process table in the set of candidate process tables is greater than the index value of the source table and less than or equal to the index value of the target table; Alternatively, if the index value of the source table is greater than the index value of the target table, a set of candidate process tables where data anomalies occur is determined from the second production list, and the candidate process table with the smallest index value is found from the set of candidate process tables and determined as the process table where the data anomaly occurs; wherein the index value of each candidate process table in the set of candidate process tables is greater than or equal to the index value of the target table and less than the index value of the source table.

[0011] Another aspect of the present invention is to provide a data anomaly location device based on time-based point sorting and pre-statistics, including: The first module is used to determine an indicator object set from various data indicators of the target data before processing the target data of the data warehouse; wherein the indicator object set includes at least one indicator object; The second module is used to perform time embedding processing on the data table set of the target data when processing the target data, determine the data processing time of each data table in the data table set, and obtain a first production list; The third module is used to perform compound sorting on the first production list after performing index value statistics on the index objects of each data table in the data table set to obtain a second production list; The fourth module is used to determine the abnormality judgment result of the target data and locate the data processing node where the data abnormality occurs according to the statistical results of the indicator values ​​of different data tables in the second production list.

[0012] In some embodiments, the first module includes: The first unit is used to determine the product quantity of the target data as an indicator object in the indicator object set before processing the target data of the data warehouse; Or, the second unit is used to determine the product amount of the target data as an indicator object in the indicator object set before processing the target data in the data warehouse.

[0013] In some embodiments, the second module includes: A third unit is used to determine the source table, the target table and the process table of the target data as a data table set of the target data when processing the target data; The fourth unit is used to add a timestamp field to the source table, the process table and the target table; A fifth unit is used to record the system time of each data table in the timestamp field when writing data to the data table set of the target data, complete the time embedding processing, and obtain the first production list; Wherein, the source table is used to represent the data source of the indicator object; The process table is used to characterize the processing process of the indicator data; The target table is used to represent the storage location of the indicator data.

[0014] In some embodiments, the third module includes: The sixth unit is used to perform counting statistics or summing statistics on the indicator objects of each data table in the data table set according to the indicator type of the indicator object to obtain the indicator value statistical result; The seventh unit is used to add a column to the first production list to record the statistical results of the indicator values, so as to obtain the second production list.

[0015] In some embodiments, the fourth module includes: An eighth unit, configured to determine a first index value of a source table and a second index value of a target table from the second production list; A ninth unit, configured to determine that an abnormality exists in the data processing process of the second production list when the first indicator value is not equal to the second indicator value; The tenth unit is used to determine a process table where the data anomaly occurs from the second production list according to the first indicator value and the second indicator value when it is determined that there is an anomaly in the data processing process of the second production list.

[0016] In some embodiments, the tenth unit comprises: The first subunit is used for comparing the statistical results of the index values ​​of each data table in the second production list one by one when it is determined that there is an abnormality in the data processing process of the second production list, and determining the process table where the data abnormality occurs from the second production list is specifically: The second subunit is used to determine a set of candidate process tables where data anomalies occur from the second production list if the index value of the source table is less than the index value of the target table, and find a candidate process table with the smallest index value from the candidate process table set to determine it as the process table where data anomalies occur; wherein the index value of each candidate process table in the candidate process table set is greater than the index value of the source table and less than or equal to the index value of the target table; The third subunit is used to determine a set of candidate process tables where data anomalies occur from the second production list if the index value of the source table is greater than the index value of the target table, and find the candidate process table with the smallest index value from the candidate process table set to determine it as the process table where the data anomaly occurs; wherein the index value of each candidate process table in the candidate process table set is greater than or equal to the index value of the target table and less than the index value of the source table.

[0017] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0018] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0019] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0020] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a method and device for locating data anomalies based on time-embedded point sorting and pre-statistics. Before processing the target data of the data warehouse, the scheme determines an indicator object set from various data indicators of the target data; wherein the indicator object set contains at least one indicator object; when processing the target data, the data table set of the target data is subjected to time-embedded point processing, the data processing time of each data table in the data table set is determined, and a first production list is obtained; after the indicator value statistics of the indicator objects of each data table in the data table set are performed, the first production list is compound sorted to obtain a second production list; according to the indicator value statistics of different data tables in the second production list, the abnormal judgment result of the target data is determined and the data processing node where the data anomaly occurs is located. The embodiments of the present invention can efficiently locate the processing node of the data anomaly from the data processing chain of the data warehouse, thereby reducing the labor cost and the time cost of production operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention; Figure 2 is a flow chart of overall steps provided by an embodiment of the present invention; Figure 3 It is a specific implementation process diagram provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0023] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0024] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0026] The data anomaly positioning method and device based on time-based point sorting and pre-statistics provided by the embodiment of the present invention relate to the field of computer technology. The data anomaly positioning method based on time-based point sorting and pre-statistics provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a data anomaly positioning method based on time-based point sorting and pre-statistics, etc., but is not limited to the above forms.

[0027] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0028] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0029] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0030] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0031] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0032] Based on the example Figure 1In the implementation environment shown, an embodiment of the present invention provides a method for locating data anomalies based on time-based point sorting and pre-statistics. The following is explained using the example of applying the method for locating data anomalies based on time-based point sorting and pre-statistics in server 101. It can be understood that the method can also be applied to terminal 102.

[0033] Reference Figure 2 , Figure 2 A flowchart of a method for locating data anomalies based on time-based point sorting and pre-statistics applied to a server provided in an embodiment of the present invention. The execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method may include the following steps: S210, before processing the target data of the data warehouse, determining an indicator object set from various data indicators of the target data; wherein the indicator object set includes at least one indicator object; Among them, data warehouse, the English name is Data Warehouse, which can be abbreviated as DW or DWH. Data warehouse is a strategic collection that provides all types of data support for decision-making processes at all levels of the enterprise. It is a single data storage created for analytical reporting and decision support purposes. It provides guidance for business process improvement, monitoring time, cost, quality and control for enterprises that need business intelligence.

[0034] In an embodiment of the present invention, a method for locating data anomalies based on time-based point sorting and pre-statistics is proposed to address the problem that it is difficult to locate abnormal data in the relevant data processing of the data warehouse. The embodiment of the present invention performs corresponding processing and analysis on the target data to be detected for abnormalities in the data warehouse. It can be understood that the target data can be a set of data for a production line process or a specific indicator in a set of data. The scope of the target data is not limited here.

[0035] Taking a set of data of a production line process as target data as an example, the target data can generally include multiple indicators. For example, for the smart home industry, the target data can be the product data of a smart home product. This product data can include different indicators such as product quantity, product amount and product qualification rate. The method of the embodiment of the present invention can perform anomaly identification for each specific indicator, or for a set of indicators, which is not limited here.

[0036] S220, when processing the target data, performing time embedding processing on the data table set of the target data, determining the data processing time of each data table in the data table set, and obtaining a first production list; Specifically, the data table set in the target data may include source tables, process tables, target tables and other data tables, each of which is used to represent the data processing stage of the target data in the data warehouse. For example, the source table may represent the starting stage of the target data, the process table may represent the intermediate processing process of the target data, and the target table may represent the processing end stage of the target data. The types of data tables are not limited here.

[0037] In addition, for the target data, the data processing chain in the entire data warehouse may involve different processing stages such as data writing, data analysis, data modification, and data output. Therefore, in order to accurately locate the time nodes and error nodes of abnormal data, the embodiment of the present invention first performs time embedding processing on the target data when data processing is about to begin. Among them, the time embedding processing can, for example, obtain the system timestamp of the current moment in real time, and use the real-time system timestamp as the embedding time of the target data.

[0038] It is understandable that in addition to using the system timestamp as the time of burying the target data, in other embodiments, time burying can be performed for each data table of the target data in other ways. For example, the user can pre-set the starting working time of the production line according to the equipment startup time of the factory production line, and use this starting working time as the time burying result of the data writing stage of the target data. Based on this starting working time, the subsequent data tables of the target data can continue to be processed accordingly. For example, when the data processing of the target data is completed after 1 hour, the starting working time plus 1 hour can be used as the burying time of the target table of the target data.

[0039] S230, after performing index value statistics on the index objects of each data table in the data table set, performing compound sorting on the first production list to obtain a second production list; Specifically, the indicator value statistics described in the embodiment of the present invention may include sum statistics of quantity, or count statistics of the number of products, etc., which is determined according to the actual needs of the application scenario and is not limited here.

[0040] In addition, after the embodiment of the present invention collects statistics on the indicator values, a data table column may be added to the first production list to specifically record the newly added indicator values. Based on this, the first production list may be sorted to obtain a second production list.

[0041] S240. Determine the abnormality judgment result of the target data according to the statistical results of the indicator values ​​of different data tables in the second production list and locate the data processing node where the data abnormality occurs.

[0042] Specifically, the embodiment of the present invention can determine whether the data tables of each stage are abnormal by comparing the index values ​​of the data tables of different data processing stages. It can be understood that for the target data of an assembly line or production line, the index value data of the source table input into the production system should be equal to the index value data of the target table output after final processing. In other words, if it is found that the data output by the target table is not equal to the source table, it means that there is an abnormality in the middle, and then the positioning of the abnormal data can be started. Based on this principle, the embodiment of the present invention can determine whether the current target data is abnormal by comparing the index values ​​of different data tables. In addition, for the target data that is confirmed to be abnormal, the process table where the abnormality occurs can be found by comparing the size difference of the index value, and then the processing node where the target data is abnormal can be determined. At this time, the positioning of the data processing node with data abnormality is completed.

[0043] In some embodiments, the step S210, before processing the target data in the data warehouse, determining the indicator object set from various data indicators of the target data, includes the following steps: S211, before processing the target data in the data warehouse, determining the product quantity of the target data as an indicator object in the indicator object set; Or, S212, before processing the target data in the data warehouse, determining the product amount of the target data as an indicator object in the indicator object set.

[0044] Specifically, the embodiment of the present invention may determine the product quantity or product amount as an indicator object in the indicator object set for subsequent data anomaly determination.

[0045] In some embodiments, the above step S220, when processing the target data, performs time embedding processing on the data table set of the target data, determines the data processing time of each data table in the data table set, and obtains the first production list, includes the following steps: S221. When processing the target data, determine the source table, target table, and process table of the target data as a data table set of the target data; Specifically, the embodiment of the present invention determines the source table, the target table and the process table as the data table set of the target data, wherein the process table can determine the amount of data according to the needs of the specific application scenario. For example, the processing process of some data warehouses may include different stages such as data preprocessing, data normalization, and data cleaning, and each stage may correspond to a process table. For this application scenario, the target data may include three process tables, a source table and a target table.

[0046] S222, adding a timestamp field to the source table, the process table and the target table; S223. When writing data to the data table set of the target data, the system time of each data table is recorded in the timestamp field, the time point embedding processing is completed, and the first production list is obtained; Wherein, the source table is used to represent the data source of the indicator object; The process table is used to characterize the processing process of the indicator data; The target table is used to represent the storage location of the indicator data.

[0047] In some embodiments, the step S223, when writing data to the data table set of the target data, records the system time of each data table in the timestamp field, completes the time embedding processing, and obtains the first production list, specifically: The system time recorded in the timestamp field of each of the data tables is sorted, and then each of the data tables is sorted according to the sorting result of the system time to obtain the first production list.

[0048] It should be noted that the ordering of system time in the embodiment of the present invention may be in ascending or descending order of time, which is not limited here.

[0049] In some embodiments, the step S230, after performing index value statistics on the index objects of each data table in the data table set, performing compound sorting on the first production list to obtain the second production list, includes the following steps: S231, performing counting statistics or summing statistics on the indicator objects of each data table in the data table set according to the indicator type of the indicator object, to obtain an indicator value statistical result; S232. Add a column to the first production list to record the statistical results of the indicator values, and obtain the second production list.

[0050] In some embodiments, the step S240, determining the abnormality determination result of the target data and locating the data processing node where the data abnormality occurs according to the statistical results of the indicator values ​​of different data tables in the second production list, includes the following steps: S241, determining a first index value of a source table and a second index value of a target table from the second production list; S242: When the first index value is not equal to the second index value, it is determined that an abnormality exists in the data processing process of the second production list; It should be noted that, for the target data of an assembly line or production line, the index value data of the source table input into the production system should be equal to the index value data of the target table output after final processing, that is, if it is found that the data output by the target table is not equal to the source table, it means that an abnormality has occurred in the middle, and the location of the abnormal data can be started. Therefore, when the embodiment of the present invention determines that the first index value is not equal to the second index value, it means that the data of the specific index object has changed before and after entering and leaving the data warehouse, which proves that the data has an abnormality, and thus determines the abnormal analysis result.

[0051] S243: When it is determined that an abnormality exists in the data processing process of the second production list, a process table where the data abnormality occurs is determined from the second production list according to the first indicator value and the second indicator value.

[0052] In some embodiments, the step S243, when it is determined that there is an abnormality in the data processing process of the second production list, determining a process table where the data abnormality occurs from the second production list according to the first indicator value and the second indicator value, includes the following steps: S2431. When it is determined that there is an abnormality in the data processing process of the second production list, the statistical results of the index values ​​of each data table in the second production list are compared one by one, and the process table where the data abnormality occurs is determined from the second production list: S2432. If the index value of the source table is smaller than the index value of the target table, a set of candidate process tables where data anomalies occur is determined from the second production list, and a candidate process table with the smallest index value is found from the set of candidate process tables to be determined as the process table where data anomalies occur; wherein the index value of each candidate process table in the set of candidate process tables is greater than the index value of the source table and less than or equal to the index value of the target table; S2433. Alternatively, if the index value of the source table is greater than the index value of the target table, a set of candidate process tables where data anomalies occur is determined from the second production list, and the candidate process table with the smallest index value is found from the set of candidate process tables to be determined as the process table where data anomalies occur; wherein the index value of each candidate process table in the set of candidate process tables is greater than or equal to the index value of the target table and less than the index value of the source table.

[0053] The following is an example of a specific application scenario. Figure 3 As shown, the specific implementation process of the present invention is described in detail: 1. Before processing the data in the data warehouse, select data indicator A as the object of data anomaly judgment. The embodiment of the present invention first performs time embedding on the source table, process table, and target table, and timestamps the data processing process. Specifically, the embodiment of the present invention can add a timestamp field when writing data to the source table, process table, and target table, and the timestamp value is the system time when the data is written.

[0054] Among them, the source table is the data source of data indicator A; the target table is the table where data indicator A is finally stored; the process table is a table formed by the processing process from the source table to the target table. There are as many process tables as there are processing processes.

[0055] 2. Set the calculation scope and statistical code of the data indicator A of the source table, process table, and target table. Exemplarily, in the embodiment of the present invention, for example, when the data indicator A is quantity, use count(A); when the data indicator is a numerical value such as amount, product quantity, data volume, etc., use sum(A). After the data is processed, execute the statistical code to count the data indicator A of the source table, process table, and target table.

[0056] In the embodiment of the present invention, the index value obtained from the source table is A1, the index obtained from the target table is B1, and the index obtained from the process table is Cn (there are as many Cn values ​​as there are process tables).

[0057] It should be noted here that for the data of an assembly line or production line, the indicator value data of the source table input into the system should be equal to the indicator value data of the target table after final processing and output. In other words, if it is found that the data output by the target table is not equal to the source table, it means that an abnormality has occurred in the middle, and you can start to locate the abnormal data.

[0058] 3. Sort the source table, process table, and target table according to the time embedding value (the value of the timestamp field) to generate the first production list L1 (L1 contains the table name and timestamp value, and the sorting is based on the timestamp). The sorting example is shown in Table 1: Table 1

[0059] 4. Integrate the index values ​​A1, B1, Cn and the time tracking value to perform compound sorting to generate the second production list L2 (L2 contains the table name, timestamp value, and index value, and the sorting is based on the index value and timestamp). The sorting example is shown in Table 2: Table 2

[0060] 5. Compare A1 and B1 in list L2. When A1=B1, the data is normal, otherwise it is abnormal.

[0061] 6. When there is an abnormality in the data, compare the data in list L2, and perform the comparison one by one in sequence: When A1 < B1, find the first process table where A1 < the list value <= B1; When A1 > B1, find the first process table where A1 > the list value >= B1.

[0062] It should be noted that the method of locating abnormal data here exemplifies a way of data comparison to achieve, and other ways can also be expanded. For example, as long as the abnormal value can be found, the abnormal data can be determined, and no limitation is made here.

[0063] 7. Determine the found process table as the data abnormality point.

[0064] 8. Exclude the location, output the processing code of the process table, and determine the cause of the problem.

[0065] In summary, in view of the technical problems in the big data scenario of the prior art, where the data processing link is long, from discovering the problem to locating the problem, it is necessary to trace back one by one along the data processing link from the discovery point, which not only takes a long time but also delays data usage and affects business operations. Through the method of the present invention, the manual input can be reduced, the abnormal patterns in the data can be automatically identified, the problem can be automatically located, the efficiency of problem-solving can be improved, and the impact on production operations can be reduced.

[0066] Another aspect of the embodiments of the present invention further provides a data abnormality positioning device based on time stamping sorting and pre-statistics, including: The first module is used to determine a set of index objects from each data index of the target data before data processing of the target data in the data warehouse; wherein, the set of index objects includes at least one index object; The second module is used to perform time stamping processing on the data table set of the target data when performing data processing on the target data, determine the data processing time of each data table in the data table set, and obtain a first production list; The third module is used to perform index value statistics on the index objects of each data table in the data table set, and then perform composite sorting on the first production list to obtain a second production list; The fourth module is used to determine the abnormal determination result of the target data and locate the data processing node where data abnormality occurs according to the index value statistical results of different data tables in the second production list.

[0067] In some embodiments, the first module includes: The first unit is used to determine the product quantity of the target data as an indicator object in the indicator object set before processing the target data of the data warehouse; Or, the second unit is used to determine the product amount of the target data as an indicator object in the indicator object set before processing the target data in the data warehouse.

[0068] In some embodiments, the second module includes: A third unit is used to determine the source table, the target table and the process table of the target data as a data table set of the target data when processing the target data; The fourth unit is used to add a timestamp field to the source table, the process table and the target table; A fifth unit is used to record the system time of each data table in the timestamp field when writing data to the data table set of the target data, complete the time embedding processing, and obtain the first production list; Wherein, the source table is used to represent the data source of the indicator object; The process table is used to characterize the processing process of the indicator data; The target table is used to represent the storage location of the indicator data.

[0069] In some embodiments, the third module includes: The sixth unit is used to perform counting statistics or summing statistics on the indicator objects of each data table in the data table set according to the indicator type of the indicator object to obtain the indicator value statistical result; The seventh unit is used to add a column to the first production list to record the statistical results of the indicator values, so as to obtain the second production list.

[0070] In some embodiments, the fourth module includes: An eighth unit, configured to determine a first index value of a source table and a second index value of a target table from the second production list; A ninth unit, configured to determine that an abnormality exists in the data processing process of the second production list when the first indicator value is not equal to the second indicator value; The tenth unit is used to determine a process table where the data anomaly occurs from the second production list according to the first indicator value and the second indicator value when it is determined that there is an anomaly in the data processing process of the second production list.

[0071] In some embodiments, the tenth unit comprises: The first subunit is used for comparing the statistical results of the index values ​​of each data table in the second production list one by one when it is determined that there is an abnormality in the data processing process of the second production list, and determining the process table where the data abnormality occurs from the second production list is specifically: The second subunit is used to determine a set of candidate process tables where data anomalies occur from the second production list if the index value of the source table is less than the index value of the target table, and find a candidate process table with the smallest index value from the candidate process table set to determine it as the process table where data anomalies occur; wherein the index value of each candidate process table in the candidate process table set is greater than the index value of the source table and less than or equal to the index value of the target table; The third subunit is used to determine a set of candidate process tables where data anomalies occur from the second production list if the index value of the source table is greater than the index value of the target table, and find the candidate process table with the smallest index value from the candidate process table set to determine it as the process table where the data anomaly occurs; wherein the index value of each candidate process table in the candidate process table set is greater than or equal to the index value of the target table and less than the index value of the source table.

[0072] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0073] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned data anomaly location method based on time-based point sorting and pre-statistics when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0074] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0075] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 401 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention; The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the data anomaly location method based on time point sorting and pre-statistics of the embodiment of the present invention; Input / output interface 403, used to implement information input and output; Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 405 that transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404); The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0076] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned data anomaly location method based on time point sorting and pre-statistics.

[0077] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0078] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0079] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0080] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0081] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0082] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0084] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0085] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0086] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0087] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0088] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0089] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs. The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A data anomaly location method based on time-based point sorting and pre-statistics, characterized in that: The following steps are involved: Before processing the target data of the data warehouse, determining an indicator object set from various data indicators of the target data; wherein the indicator object set includes at least one indicator object; When processing the target data, time-point processing is performed on the data table set of the target data to determine the data processing time of each data table in the data table set, and obtain a first production list; After performing index value statistics on the index objects of each data table in the data table set, performing compound sorting on the first production list to obtain a second production list; According to the statistical results of the indicator values ​​of different data tables in the second production list, the abnormality judgment result of the target data is determined and the data processing node where the data abnormality occurs is located.

2. According to claim 1, a method for locating data anomalies based on time-based point sorting and pre-statistics is characterized in that: Before processing the target data in the data warehouse, determining the indicator object set from various data indicators of the target data includes the following steps: Before data processing is performed on the target data of the data warehouse, the product quantity of the target data is determined as an indicator object in the indicator object set; Or, before processing the target data in the data warehouse, the product amount of the target data is determined as an indicator object in the indicator object set.

3. According to the method of claim 1, the method is characterized in that: When processing the target data, performing time embedding processing on the data table set of the target data, determining the data processing time of each data table in the data table set, and obtaining a first production list, including the following steps: When processing the target data, a source table, a target table and a process table of the target data are determined as a data table set of the target data; For the source table, the process table and the target table, a timestamp field is added; When writing data to the data table set of the target data, the system time of each data table is recorded in the timestamp field, the time embedding processing is completed, and the first production list is obtained; Wherein, the source table is used to represent the data source of the indicator object; The process table is used to characterize the processing process of the indicator data; The target table is used to represent the storage location of the indicator data.

4. According to claim 3, a method for locating data anomalies based on time-based point sorting and pre-statistics is characterized in that: When writing data to the data table set of the target data, the system time of each data table is recorded in the timestamp field, the time embedding processing is completed, and the first production list is obtained, which is specifically: The system time recorded in the timestamp field of each of the data tables is sorted, and then each of the data tables is sorted according to the sorting result of the system time to obtain the first production list.

5. According to claim 1, a method for locating data anomalies based on time-based point sorting and pre-statistics is characterized in that: After the indicator value statistics are performed on the indicator objects of each data table in the data table set, the first production list is compound sorted to obtain the second production list, including the following steps: According to the indicator type of the indicator object, counting statistics or summing statistics are performed on the indicator objects of each data table in the data table set to obtain an indicator value statistical result; A column is added to the first production list to record the statistical results of the indicator values, so as to obtain the second production list.

6. According to claim 1, a method for locating data anomalies based on time-based point sorting and pre-statistics is characterized in that: The step of determining the abnormality determination result of the target data and locating the data processing node where the data abnormality occurs according to the statistical results of the indicator values ​​of different data tables in the second production list comprises the following steps: Determine a first index value of a source table and a second index value of a target table from the second production list; When the first index value is not equal to the second index value, it is determined that an abnormality exists in the data processing process of the second production list; When it is determined that an abnormality exists in the data processing process of the second production list, a process table where the data abnormality occurs is determined from the second production list according to the first indicator value and the second indicator value.

7. According to claim 6, a method for locating data anomalies based on time-based point sorting and pre-statistics is characterized in that: When it is determined that there is an abnormality in the data processing process of the second production list, determining a process table where the data abnormality occurs from the second production list according to the first indicator value and the second indicator value, comprises the following steps: When it is determined that there is an abnormality in the data processing process of the second production list, the statistical results of the index values ​​of each data table in the second production list are compared one by one, and the process table where the data abnormality occurs is determined from the second production list: If the index value of the source table is less than the index value of the target table, a set of candidate process tables where data anomalies occur is determined from the second production list, and a candidate process table with the smallest index value is found from the set of candidate process tables to be determined as the process table where data anomalies occur; wherein the index value of each candidate process table in the set of candidate process tables is greater than the index value of the source table and less than or equal to the index value of the target table; Alternatively, if the index value of the source table is greater than the index value of the target table, a set of candidate process tables where data anomalies occur is determined from the second production list, and the candidate process table with the smallest index value is found from the set of candidate process tables and determined as the process table where the data anomaly occurs; wherein the index value of each candidate process table in the set of candidate process tables is greater than or equal to the index value of the target table and less than the index value of the source table.

8. A data anomaly locating device based on time-based point sorting and pre-statistics, characterized in that: include: The first module is used to determine an indicator object set from various data indicators of the target data before processing the target data of the data warehouse; wherein the indicator object set includes at least one indicator object; The second module is used to perform time embedding processing on the data table set of the target data when processing the target data, determine the data processing time of each data table in the data table set, and obtain a first production list; The third module is used to perform compound sorting on the first production list after performing index value statistics on the index objects of each data table in the data table set to obtain a second production list; The fourth module is used to determine the abnormality judgment result of the target data and locate the data processing node where the data abnormality occurs according to the statistical results of the indicator values ​​of different data tables in the second production list.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images