A data detection method, device, apparatus and storage medium

Abstract

The application provides a data detection method and device, equipment and a storage medium, wherein the method comprises: querying data of a distributed application server node, and performing format processing on the queried data to obtain a data model object conforming to a predefined data model; shunting the data model object to a corresponding channel for aggregation to obtain an aggregated data sequence pair, and collecting each data model object associated with the aggregated data sequence pair in a data stream; setting a dynamic data window range based on a starting time of each data model object entering the data stream and an end time of each data model object in the data stream; and processing the aggregated data sequence pair in the data stream based on the set dynamic data window range to generate a detection result for the queried data, wherein the detection result comprises whether the queried data is consistent with expected data.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing in financial technology (Fintech), and relate to but are not limited to a data detection method, device, equipment and storage medium. BACKGROUND

[0002] With the development of computer computing, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, higher requirements are also put forward for technology.

[0003] In the field of financial technology, current projects are mostly deployed under a distributed architecture, and application and database instances are distributed in multiple node regions, and nodes communicate with each other through corresponding services. The current data consistency detection scheme is applied to a relational database management system such as Mysql master-slave architecture design, relies on Mysql to open a file (such as a binary file Binlog) for synchronization, generates a checksum of a data block in a master library execution mode, and transmits the Binlog synchronization log to a slave library execution mode. After execution, the same checksum of the data block is calculated in the slave library. This data consistency detection scheme relies on the Binlog synchronization implementation of the master-slave architecture of the Mysql database. When multiple database types exist simultaneously in a distributed cross-domain scenario, for example, in a scenario where multiple different types of database relationship systems exist in a distributed architecture design: Mysql, TiDB, Oracle, because various types of databases cannot form a master-slave architecture, they cannot support detection in a distributed multi-database type architecture scenario. Based on this, related technologies cannot support cross-domain data consistency detection in the case of cross-domain and cross-database types. SUMMARY

[0004] Embodiments of the present application provide a data detection method, device, equipment and storage medium to solve the problem that existing technologies cannot support cross-domain data consistency detection in the case of cross-domain and cross-database types.

[0005] The technical scheme of the embodiments of the present application is as follows:

[0006] The embodiments of the present application provide a data detection method, comprising:

[0007] querying the data of the distributed application server node, and formatting the queried data to obtain a data model object conforming to a predefined data model;

[0008] The data model objects are shunted into corresponding channels for aggregation to obtain aggregated data sequence pairs, and each data model object associated with the aggregated data sequence pairs is collected into a data stream;

[0009] Based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream, a dynamic data window range is set;

[0010] Based on the set dynamic data window range, the aggregated data sequence pairs in the data stream are processed to generate a detection result for the queried data; wherein the detection result includes whether the queried data is consistent with expected data.

[0011] A data detection device, the device comprises:

[0012] A preprocessing module is configured to query data of a distributed application server node and format the queried data to obtain data model objects conforming to a predefined data model;

[0013] An aggregation module is configured to shunt the data model objects into corresponding channels for aggregation to obtain aggregated data sequence pairs, and collect each data model object associated with the aggregated data sequence pairs into a data stream;

[0014] A window setting module is configured to set a dynamic data window range based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream;

[0015] A detection result matching module is configured to process the aggregated data sequence pairs in the data stream based on the set dynamic data window range to generate a detection result for the queried data; wherein the detection result includes whether the queried data is consistent with expected data.

[0016] A data detection device, comprising:

[0017] A memory is configured to store executable instructions; and a processor is configured to execute the executable instructions stored in the memory to implement the above method.

[0018] A computer readable storage medium stores executable instructions for causing a processor to execute the above method when the executable instructions are executed.

[0019] The embodiments of the present application have the following beneficial effects:

[0020] The data of the distributed application server node is queried, and the queried data is formatted to obtain a data model object conforming to a predefined data model. That is, the data detection method provided in the application can process the data of the distributed application server node, that is, the data of different data sources, and format the data of different data sources, thereby supporting cross-domain and cross-database data detection. Further, the data model object is distributed to a corresponding channel for aggregation to obtain an aggregated data sequence pair, and each data model object associated with the aggregated data sequence pair is collected into a data stream. Based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream, a dynamic data window range is set. Based on the set dynamic data window range, the aggregated data sequence pair in the data stream is processed to generate a detection result for the queried data. The detection result includes whether the queried data is consistent with expected data. Here, the appropriate data range is framed based on the set dynamic data window range for detection, which not only ensures the integrity of the data, but also improves the access benchmark of the data volume based on the stream computing of the dynamic data window range, which can process a large amount of data flow and improve the detection efficiency. In this way, the data detection method provided in the application achieves the purpose of supporting cross-domain data consistency detection and improving the detection efficiency in the case of cross-domain and cross-database types. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is an optional architecture schematic diagram of a terminal provided in an embodiment of the application;

[0022] Figure 2 is a flowchart of a data detection method provided in an embodiment of the application Figure 1 ;

[0023] Figure 3 is a flowchart of a data detection method provided in an embodiment of the application Figure 2 ;

[0024] Figure 4 is a data flow direction schematic diagram provided in an embodiment of the application;

[0025] Figure 5 is a flowchart of a data detection method provided in an embodiment of the application Figure 3 ;

[0026] Figure 6 is a schematic diagram of a dual-time-dimension-defined time range provided in an embodiment of the application;

[0027] Figure 7 is a schematic diagram of determining the maximum value in the query interval provided in an embodiment of the application;

[0028] Figure 8 This is a schematic diagram of streaming window computation provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0031] The following describes exemplary applications of the data detection device provided in this application. This data detection device can be implemented as any terminal with a screen display function, such as a laptop, tablet, desktop computer, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), or intelligent robot, or as a server. The following will describe exemplary applications when the data detection device is implemented as a terminal.

[0032] See Figure 1 , Figure 1 This is a schematic diagram of the structure of the terminal 100 provided in the embodiments of this application. Figure 1 The terminal 100 shown includes at least one processor 110, at least one network interface 120, a user interface 130, and a memory 150. The various components in the terminal 100 are coupled together via a bus system 140. It is understood that the bus system 140 is used to implement communication between these components. In addition to a data bus, the bus system 140 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 1 The general labeled all buses as Bus System 140.

[0033] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0034] User interface 130 includes one or more output devices 131 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 130 also includes one or more input devices 132, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0035] Memory 150 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 150 may optionally include one or more storage devices physically located remote from processor 110. Memory 150 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 150 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.

[0036] Operating system 151 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0037] The network communication module 152 is used to reach other computing devices via one or more (wired or wireless) network interfaces 120, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0038] The input processing module 153 is used to detect and translate one or more user inputs or interactions from one or more input devices 132.

[0039] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 1 A data detection device 154 stored in memory 150 is shown. This data detection device 154 can be a data detection device in terminal 100, and can be software in the form of programs and plug-ins, including the following software modules: preprocessing module 1541, aggregation module 1542, window setting module 1543, and detection result matching module 1544. These modules are logically connected, and therefore can be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0040] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data detection method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0041] This section further explains current data consistency detection schemes. In scenarios where strong data consistency is required in the transaction chain, transactions can typically guarantee data consistency. However, for some abnormal scenarios that cannot be guaranteed programmatically, such as application service downtime, manual data manipulation, data migration (DM) synchronization conflicts, and master-slave synchronization inconsistencies, data inconsistencies can occur between data center nodes (DCNs) in the production environment, leading to data conflicts and data fragmentation. Currently, the industry typically uses a collection of advanced command-line tools (Percona-Toolkit, PT) to implement MySQL master-slave database consistency detection.

[0042] As mentioned earlier, in distributed cross-domain scenarios where multiple database types coexist, such as MySQL, TiDB, and Oracle in a distributed architecture design, the detection of distributed multi-database type architecture scenarios cannot be supported because the various types of databases cannot form a master-slave architecture.

[0043] Furthermore, the detection process is limited to MySQL master-slave architectures, requiring Binlog synchronization to be enabled, and the table structure to be consistent. Only one table can be processed at a time. The reasons are as follows: Binlog synchronization inevitably involves considering master-slave synchronization latency, which is highly likely to occur when synchronizing large amounts of data. Therefore, the current data consistency detection scheme needs to control the synchronization rate and the amount of data synchronized, thus limiting the detection to a single table at a time. The table is split into row blocks, and each block's master-slave node data is tested. This directly compromises efficiency and fails to decouple the dependency from the underlying database characteristics.

[0044] Therefore, this application provides a data detection method that supports data consistency detection and improves detection efficiency in cross-domain and cross-database types.

[0045] The data detection method provided in this application will be described below with reference to exemplary applications and implementations of the terminal 100 provided in the embodiments of this application. See also Figure 2 , Figure 2 This is an optional flowchart illustrating the data detection method provided in the embodiments of this application, which will be combined with... Figure 2 The steps shown are explained below.

[0046] Step S201: Query the data of the distributed application server node, and format the queried data to obtain a data model object that conforms to the predefined data model.

[0047] The data detection method provided in this application is applicable to scenarios including, but not limited to, combinations of one or more of the following: heterogeneous database scenarios, and multiple data sources coexisting in a distributed cross-database complex architecture scenario.

[0048] In this embodiment, a unified data model (MapMode) protocol is defined to perform unified mapping and encapsulation processing on data in the dataset to be consistent, thereby implementing a formatting process. When defining MapMode, the parameters involved include some or all of the following: data node, data type, data entity, data identity information, and data cursor. Here, these parameters are also called attributes, and therefore, the parameter values ​​are also called attribute values. Specifically, the attribute value corresponding to the data node is used to store the DCN node information of the data source, such as the DCN region corresponding to the data; the attribute value corresponding to the data type is used to store the data type information, such as the order table type; the attribute value corresponding to the data entity is used to store the retrieved entity data information, such as {field1: data1, field2: data2}; the attribute value corresponding to the data identity information is used to store the primary key information of the retrieved data; and the attribute value corresponding to the data cursor is used to store the data cursor position information.

[0049] In some embodiments of this application, taking the defined MapMode involving data nodes, data types, data entities, data identity information, and data cursors as examples, the data from the queried distributed application server nodes is combined into a MapMode data packet, wherein the MapMode data packet includes parameters as shown in Table 1:

[0050] Attribute Attribute Name Attribute Value dataNode Data Node DCN Area Datatype Data Type order Order Table Type dataEntity Data Entity {Field 1: Data 1, Field 2: Data 2} dataIdentity Data Identity Information Primary Key Information dataVernier Data Vernier 1

[0051] Table 1. Parameters included in the MapMode data packet

[0052] In a feasible data preprocessing scenario, when a client on a terminal queries data on a distributed application server node, it can use a data cursor (dataVernier) to query the data sequentially in segments. The queried data is then formatted and mapped and encapsulated into a MapMode object as defined above. This MapMode object is transmitted based on structured data storage methods, such as the serialization method for structured data (Protocol Buffer, Protobuf). The client can be a data sentinel (WatchDog) client integrated and deployed on the terminal. In other words, in this application, a data sentinel (WatchDog) client can be integrated and deployed on each distributed application server node, and data detection processing can be performed based on the input / output (IO) thread model of an open-source framework such as Netty to improve the concurrency and speed of data transmission.

[0053] For example, after data collection and processing via the WatchDog client, the MapMode data mapping is as follows:

[0054] The dataNode stores data segment DCN node information;

[0055] dataType stores information about the data type;

[0056] The entity data information retrieved by dataEntity;

[0057] The primary key information of the data retrieved by dataIdentity;

[0058] DataVernier cursor position information;

[0059] The following is a MapMode object obtained after data collection and processing via the WatchDog client and mapping using MapMode:

[0060]

[0061] In this embodiment, the client can be an application running on a terminal or a web application loaded in a webpage.

[0062] Step S202: The data model objects are distributed to the corresponding channels for aggregation to obtain aggregated data sequence pairs, and the data model objects associated with the aggregated data sequence pairs are gathered into the data stream.

[0063] In this embodiment, the collected data set is serialized into a MapMode data model, and the data model objects are distributed to the corresponding channels for aggregation to obtain aggregated data sequence pairs. The data model objects associated with the aggregated data sequence pairs are then gathered into the data stream. This ensures that the streaming computation based on the dynamic data window range is oriented towards an infinite input data stream, thereby improving the data volume threshold, enabling the processing of large amounts of data, and improving detection efficiency.

[0064] Step S203: Set the dynamic data window range based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream.

[0065] The data window is the time-limited window of data within the data stream.

[0066] In this embodiment, the dynamic data window range is a data range determined at least based on two time dimensions: the start time of each data model object entering the data stream and the end time of each data model object in the data stream. This avoids the problem of inaccurate data range in a single time dimension, and the above-mentioned dual time dimension limitation of the data range has higher accuracy.

[0067] Step S204: Based on the set dynamic data window range, process the aggregated data sequence pairs in the data stream to generate detection results for the queried data.

[0068] The detection results include whether the retrieved data matches the expected data.

[0069] In this embodiment, based on a set dynamic data window range, aggregated data sequence pairs in the data stream are processed to generate a detection result indicating whether the queried data matches the expected data. Furthermore, based on different detection results, event pattern matching (EventMode) can be triggered to notify the data detection results.

[0070] The data detection method provided in this application queries data from distributed application server nodes and formats the queried data to obtain data model objects that conform to a predefined data model. In other words, the data detection method provided in this application can process data from distributed application server nodes, i.e., data from different data sources, and format the data from different data sources, supporting cross-domain and cross-database data detection. Furthermore, the data model objects are distributed to corresponding channels for aggregation, resulting in aggregated data sequence pairs. The data model objects associated with each aggregated data sequence pair are then gathered into a data stream. Based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream... In this process, a dynamic data window range is set. Based on the set dynamic data window range, the aggregated data sequence pairs in the data stream are processed to generate detection results for the queried data. The detection results include whether the queried data is consistent with the expected data. Here, defining an appropriate data range based on the set dynamic data window range for detection not only ensures data integrity, but also, since streaming computation based on the dynamic data window range is oriented towards an infinite input data stream, it raises the threshold for data volume entry, can handle large amounts of data flow, and improves detection efficiency. Thus, the data detection method provided in this application achieves the goal of supporting cross-domain data consistency detection and improving detection efficiency in cross-domain and cross-database types.

[0071] In some embodiments of this application, in step S202, the data model object is distributed to the corresponding channel for aggregation to obtain aggregated data sequence pairs. This can be achieved through methods such as... Figure 3 The steps shown are to be implemented as follows:

[0072] Step S2021: Obtain the data sequence pairs of the data model object.

[0073] The data sequence pair includes a data entity feature sequence and a data information feature sequence.

[0074] Here, the data entity feature sequence includes the features of the data entities contained in the formatted data after the query is completed; the data information feature sequence includes the features of the data collection time, the data nodes contained in the formatted data, the data tables, etc., that are present in the query data.

[0075] Step S2022: Determine the queue corresponding to the feature values ​​contained in the feature sequence of the data entity.

[0076] In this embodiment, identical feature values ​​indicate identical data entities. There is a one-to-one correspondence between feature values ​​and queues; that is, data corresponding to the same feature value selects the same queue during transmission, thus improving the concurrency and speed of data transmission.

[0077] Step S2023: Based on at least one determined queue, the data model objects are distributed to the corresponding channels for aggregation to obtain aggregated data sequence pairs.

[0078] In this embodiment, the collected data set is serialized into a MapMode data model. Then, the data undergoes time-series processing and feature sequence pair extraction to ensure the data's temporal order and that the same dataset flows to the same queue channel. Subsequently, a channel selector polls the data queues to determine which queue channel the data model will be distributed to for aggregation.

[0079] In some embodiments of this application, obtaining the data sequence pairs of the data model object in step S2021 can be achieved through the following steps:

[0080] A11 retrieves the data length, product coefficient, modulo coefficient, and encoding standard value of the string mapping contained in the data model object.

[0081] In this embodiment, comparing whether two strings are equal involves comparing the substrings mapped to the two strings, rather than directly comparing the two strings, thus improving the accuracy of the comparison. Here, the substrings mapped to the strings are represented as feature values. The encoding standard value, product coefficient, modulo coefficient, and data length of the string mapping are used to calculate the feature values ​​of the string mapping.

[0082] A12 determines the feature values ​​of strings based on the product coefficient, modulo coefficient, data length, and string mapping encoding standard values, and uses the feature values ​​of all strings contained in the data model object as the data entity feature sequence of the data model object.

[0083] Among them, the data entity feature sequence h i Used to determine differences in the characteristics of data entities.

[0084] In some embodiments, A12, based on the product coefficient, modulo coefficient, data length, and the encoding standard value of the string mapping, the feature value of the string is determined. This can be achieved by substituting the product coefficient, modulo coefficient, data length, and the encoding standard value of the string mapping into a hash function to perform a hash mapping on the string, thereby obtaining the string's feature value. Here, performing a hash mapping on the string refers to mapping different strings to different numbers using a certain string hash function.

[0085] In a feasible scenario, the feature values ​​of a string can be determined using the hash function shown in formula (1), thereby obtaining the feature sequence h of the data entity. i .

[0086] The above calculations using A11-A12 yield h. iThis can be achieved through the following calculation formula (1).

[0087]

[0088] Where m[j] is the character of the data entity dataEntity in MapMode, p j is the product coefficient, which can be defined as a small value; mod is the modulo coefficient, which can be defined as a large value; n is the current data length; i and j are positive integers; idx(m[j]) is the character mapping encoding standard value, such as the ASCII code value. That is, each character idx(m[j]) of the data entity dataEntity corresponds to its decimal value in the ASCII table. The ASCII table here can refer to the ASCII code lookup table in related technologies. This application does not make specific limitations on it.

[0089] In a feasible data consistency detection scenario, see Figure 4 As shown, suppose there are four distributed application server nodes (ABCD) whose data (i.e., cross-domain, cross-database DCN node data) are normalized into MapMpde objects:

[0090] Below, ABCD represent MapMpde objects generated from data processing of different DCN nodes.

[0091] A:{"dataNode":"AA0","dataType":"order","dataEntity":"{'table':'order_info','record':{'user_name':'zhangsan','s eal_type':'2','trans_status':'SUCCESS'}}","dataIdentity":"2102240QD022000A96UN7M0LI0CXUDC0","dataVernier":"1"}

[0092] B:{"dataNode":"AJ0","dataType":"order","dataEntity":"{'table':'order_info','record':{'user_name':'zhangsan','s eal_type':'2','trans_status':'SUCCESS'}}","dataIdentity":"2102240QD022000A96UN7M0LI0CXUDC0","dataVernier":"1"}

[0093] C:{"dataNode":"AK0","dataType":"order","dataEntity":"{"table":"order_info","record":{"user_name":"zhangsan", "seal_type":"2","trans_status":"FAIL"}}","dataIdentity":"2102240QD022000A96UN7M0LI0CXUDC0","dataVernier":"1"}

[0094] D:{"dataNode":"AI0","dataType":"order","dataEntity":"{"table":"order_info","record":{"user_name":"zhangsan","s eal_type":"2","trans_status":"SUCCESS"}}","dataIdentity":"2102240QD022000A96UN7M0LI0CXUDC0","dataVernier":"1"}

[0095] Among them, the normalized MapMpde objects A, B, and D have consistent data entities in their corresponding nodes (dataEntity). However, the normalized MapMpde object C has a field that has changed in its corresponding node (dataNode AK0). Furthermore, the data entities of ABCD are converted into unified data entity feature value substrings.

[0096] Furthermore, each character idx(m[j]) in the data entity dataEntity corresponds to its decimal value in the ASCII table. Assuming p = 3, mod = 81001, h[0] = 0, we iterate through each string, calculate the value, sum it up, and then take the modulo.

[0097] Here, taking a string from A as an example, the calculation for "{'SUCCESS'}" is: a = ((0×3 1 +123)+(123×3 2 +39)+(1146×3 3 +83)+(31025×3 4 +85)+(h[j-1]×3 j +idx(m[j])))%81001.

[0098] Here, let's take another string from C as an example. The calculation for "{'FAIL'}" is: c = ((0×3)1 +123)+(123×3 2 +39)+(1146×3 3 +70)+(31012×3 4 +65)+(h[j-1]×3 j +idx(m[j])))%81001.

[0099] Perform h on the MapMode of ABCD i The hash operation yields an index sequence of [a,b,c,d], where a = b = d. In other words, the data entities in A, B, and C have the same feature value.

[0100] A13 obtains the source database region, data source table, and data query time of the data model object, and uses the source database region, data source table, and data query time as the data information feature sequence of the data model object.

[0101] Where, d c For the source database DCN region, t a For the data source table Table, w t The data query time is also known as the data acquisition time WatchTime (wt);

[0102] Here, h(d) c +t a +w t ) Calculate and obtain the data information feature sequence h k .

[0103] Among them, h(d) c +t a +w t ) Calculate and obtain h k This can be achieved in the following way: d c +t a +w t As a string, h is calculated using the aforementioned formula (1). k .

[0104] A14, a data sequence pair that forms a data model object based on the data entity feature sequence and the data information feature sequence.

[0105] Here, based on h i and h k After the operation, the data sequence pair w is obtained. h w h = <h i ,h k >

[0106] Still taking the MapMpde objects processed from the data of different DCN nodes represented by ABCD as an example, the data sequence pairs w calculated by ABCD h The set of is as follows: [<a, a'>, <b, b'>, <c, c'>, <d, d'>];

[0107] where a = b = d are the same, assumed to be 1001, and c is 1002.

[0108] The final data sequence pair w h is as follows:

[0109] <a{1001}, a'{v: <d c : AA0, t a : order, w t : 20220401080100}>,

[0110] <b{1001}, b'{v: <d c : AJ0, t a : order, w t : 20220401080200}>,

[0111] <c{1002}, c'{v: <d c : AK0, t a : order, w t : 20220401080300}>,

[0112] <d{1001}, d'{v: <d c : AI0, t a : order, w t : 20220401080400}>,

[0113] Furthermore, as shown in combination with Figure 4 the client routes the queue according to the data entity feature sequence h through a selector (Selector). i Since the eigenvalue of ABD is a = b = d, the same Q1 queue is selected, and C selects the Q2 queue. Then, in the form of a queue, the data model objects are shunted to the corresponding channels and sent to the data aggregation center (AggCenter).

[0114] In some embodiments of the present application, step S2023 shunts the data model objects to the corresponding channels for aggregation to obtain the aggregated data sequence pairs, which can be implemented through the following steps:

[0115] B21 creates a new temporary measurement block for the same data entity feature sequence and marks the aggregation time series and the same block color for the temporary measurement block.

[0116] In the embodiments of this application, AggCenter completes the aggregation processing of the reported MapMode data. AggCenter processes the data in the queue with the same source through the aggregator by the eigenvalue dimension data, creates a new temporary measurement block for the same data entity feature sequence, and colors this new block with the same block color. The colored block (AggBlock) is an independent granular unit. See Figure 4 As shown, the AggBlocks of different colors in the aggregation center are represented by different patterns. Figure 4 Four example patterns are given in. This block is responsible for uniformly executing status operations, improving the processing efficiency of the source data. And by simultaneously marking (tagging) and attaching the aggregation time AggTime (at), AggTime (at) is used when determining the dual-time dimension range.

[0117] At this time, the AggBlock aggregation data sequence pair of ABD is a h is a abd :

[0118] [<a{1001},a′{v:{<d c :AA0,t a :order,w t :2022040<1080100>,<tag:red,a t :2022040<1080400>}}>,

[0119] <b{1001},b′{v:{<d c :AJ0,t a ​​​​​​​​​​​​​​​​​​​​​​​

[0122] [<c{1002},c′{v:{<d c :AK0,t a :order,w t :2022040<1080300>,<tag:blue,a t :2022040<1080500>}}>,d>]

[0123] After the data is aggregated, the internal data state is flipped through the coloring process of the AggBlock block, improving the data processing efficiency. At the same time, in the data aggregation data stream, it is continuously transmitted into the data stream to form an unbounded data set.

[0124] B22. Extract the query time series, source database area, and data source data table from the data entity feature sequence of the data model object.

[0125] B23. Obtain the aggregated data sequence pair based on the eigenvalue, source database area, data source data table, query time series, block color, and aggregation time series.

[0126] In the embodiments of the present application, AggTime is concatenated to form a time axis. Through the ABCD data sequence pair w h Extract the acquisition time series: a h Extract the aggregation time series from: Among them,

[0127] w t =[20220401080100, 20220401080200, 20220401080300, 20220401080400];

[0128] a t =[20220401080400, 20220401080400, 20220401080500, 20220401080400].

[0129] In some embodiments of the present application, the start time is the data query time of each data model object, and the end time is the aggregation time of each data model object. Step S203 sets the dynamic data window range based on the start time when each data model object enters the data stream and the end time of each data model object in the data stream, which can be implemented through the steps as Figure 5 shown:

[0130] Step S2031. Obtain the set predetermined window and delay window.

[0131] The delay window is the maximum allowed delay time for data in the data stream.

[0132] In this embodiment of the application, the predetermined window WindowTime(w it ) and DelayTime(d et ) can be flexibly configured according to actual needs. Among them, w it Used to ensure the baseline range data, d et The range of data windows used to ensure delayed delivery.

[0133] For example, setting WindowTime(w it The time interval is 1 minute, and the DelayTime(d) is 1 minute. et This indicates the maximum allowed delay time for the window, which is also set to 1 minute.

[0134] Step S2032: The time range covered by the minimum and maximum time nodes included in the data query time and aggregation time is used as a dynamic adjustment window.

[0135] In this embodiment of the application, DynamicTime(d) is used as the dynamic window time. yt ) represents the dynamic window range, and d yt Through dual time dimensions [w t a t [Limited]. The data query time is the same as the data collection time w. t Let 'a' be the start time of each data point entering the data stream, and 'aggregation time' be denoted as 'a'. t Let d be the end time of each data point in the data stream, and determine d through two dimensions. yt This avoids the problem of inaccurate data range that occurs in a single time dimension, and the dual time dimension constraint has higher accuracy.

[0136] The smallest time node is d. min The maximum time point is d max , then d yt =len[d min ,d max ].

[0137] Here, based on the aforementioned determined data collection time sequence... and aggregated time series The two dual-dimensional timelines mentioned above can intersect to form a time range, as shown in the reference. Figure 6 As shown, on the intersection time axis, the range from the minimum time node to the maximum time node can be considered as the dynamic window range, that is, the time range defined by dual time dimensions. The collected time series w... t and aggregated time series a tAfter merging, we get a dynamic window time series array d. t = [t1,t2,t3,t4,t1′,t2′,t3′,t4′], and take the union of this sequence, d yt =len[d min ,d max ].

[0138] Step S2033: Set the range of the dynamic data window based on the predetermined window, the delay window, and the dynamic adjustment window.

[0139] In this embodiment of the application, the dynamic data window scope WindowScope(w s The window is determined based on the pre-defined window, the delay window, and the dynamically adjusted window.

[0140] For example, w s It can be determined by the following calculation formula (2),

[0141] w s =w it +d yt +d et Formula (2)

[0142] In this embodiment of the application, multi-dimensional dynamic settings w are adopted. s This method improves the accuracy of range selection for streaming data.

[0143] In some embodiments of this application, before step S2032 uses the time range from the minimum time node to the maximum time node covered by the data query time and aggregation time as the dynamic adjustment window, the minimum time node to the maximum time node can be determined through the following steps:

[0144] Based on the sparse table, retrieve the minimum and maximum time nodes in the data query time and aggregation time.

[0145] In this embodiment of the application, after obtaining the collection time series and aggregated time series In this case, the time series w will be collected. t and aggregated time series a t After merging, we get a dynamic window time series array d. t = [t1,t2,t3,t4,t1′,t2′,t3′,t4′], and during the process of taking the union of this sequence, the smallest time node d between the data query time and the aggregation time can be retrieved based on the sparse table. min and the maximum time node d max Here, the sparse table doubling method can be used to find the maximum and minimum values ​​within the dynamic window time series, thereby determining the maximum value of the length of the dual time dimension range.

[0146] In a feasible scenario for determining the maximum and minimum values, during the preprocessing stage, for array t, let t[i][j] represent the left endpoint and j represent 2. j A length, that is, in d t [i] is a consecutive 2 starting point j The maximum number of elements. Since the number of elements is 2. j There are 2, so if we divide it into two equal parts, each part will have 2. j-1 Therefore, t[i,j] represents the interval [i,i+2]. j-1 The maximum value within the range.

[0147] Here, using the dynamic window time series array d determined above... t For example: d t =[20220401080100,20220401080200,20220401080300,20220401080400,20220401080400,20220401080400,20220401080400,20220401080500,20220401080400]

[0148] Where t[1][0] represents the first number, with a length of 2. 0 The maximum value of 1 is actually the first value, 20220401080100.

[0149] t[1][1] indicates that the length is 2 starting from the first number. 1 The maximum value of 2 is t[1,1] = max(20220401080100,20220401080200) = 20220401080200.

[0150] t[1][2] represents the first number, with a length of 2. 2 =4, t[1,2]=max(20220401080100,20220401080200,20220401080300,20220401080400)=20220401080400.

[0151] ...

[0152] Then t[i][0] represents the maximum value of one consecutive point starting from i, i.e., [i, i].

[0153] t[i][1] represents the maximum value of two consecutive points starting from i, i.e., the maximum value in [i, i+1].

[0154] t[i][2] represents the maximum value of 4 consecutive points starting from i, i.e., the maximum value in [i, i+1, i+2, i+3].

[0155] t[i][3] represents the maximum value of 8 consecutive points starting from i, i.e., the maximum value in [i,i+1,i+2,i+3,…,i+7].

[0156] The state transition equation is expressed as the following calculation formula (3),

[0157] t[i,j]=max(t[i,j-1],t[i+2 j-1 Formula (3)

[0158] Each t[i][j] in the above preprocessing is an interval of length 2. j The maximum and minimum values ​​of the interval. (Refer to...) Figure 7 As shown, suppose the interval to be queried is [l, r]. We need to find the union of two sub-intervals that contains the entire query interval, and these two sub-intervals must be of the same length. To ensure that these two smaller intervals can contain the entire larger interval, the length of a single smaller interval must be no less than half the length of the query interval to achieve complete coverage of the query interval length. Secondly, the interval length must be a power of 2; the length of a smaller interval cannot cover the larger interval, but twice the length of the smaller interval should cover the larger interval.

[0159] The maximum value query for an interval is as follows: the interval to be queried is [l, r], the interval length is r-l+1, and it satisfies 2 k ≤r-l+1<2 k+1 We can take k = log2(r-l+1) and round it down, then the maximum time node d can be determined by formula (4). max :

[0160] d max =max(t[l,k],t[r-2) k +1][k]) Formula (4)

[0161] For example, d t [20220401080100,20220401080200,20220401080300,20220401080400,20220401080400,20220401080400,20220401080400,20220401080500,20220401080400], take the maximum value in the interval [1,8], which is k = log2(8-1+1) = 3, that is, find: max(t[1,3],t[8-2] 3 +1][3]), the maximum value is 20220401080500.

[0162] Similarly, formula (4) calculates the minimum value d in reverse using min(). min Given 20220401080100, the sequence pair d is obtained. yt =len[20220401080100,20220401080500]. The final calculation yields the dynamic adjustment window d for the ABCD time series. yt The length is 4.

[0163] Therefore, the dynamic data window range w s The size is: 1 + 4 + 1 = 6.

[0164] In some embodiments of this application, the expected data is the configuration threshold of the distributed application server node. Step S204 processes the aggregated data sequence pairs in the data stream based on the set dynamic data window range to generate detection results for the queried data. This can be achieved through the following steps:

[0165] First, operations are performed on the aggregated data sequence pairs within the dynamic data window to filter out each color block and determine the data threshold within each color block.

[0166] Secondly, within the aggregated data sequence within the dynamic data window, the feature quantity value for each color block is determined.

[0167] The feature quantity value represents the number of data with the same feature value.

[0168] Finally, based on the data threshold within each color block, the feature quantity value of each color block, and the configuration threshold, the detection results are generated.

[0169] Figure 8 In a feasible streaming window setup scenario, data tasks are processed via triggers (Window Triggers) to dynamically manage the window range w. s Aggregated data sequence pair a within the range h The calculation involves first identifying each colored block and then calculating the data threshold within that block. Here, since a has already been processed... abd ,a c The block data is identified as data with common characteristics; therefore, it is only necessary to add data to the aggregated data sequence for pair a. h The system calculates the number of features for each block and then uses this number to dynamically query the number of DCN nodes. Further, it determines whether the expected value is met and triggers an EventMode match to notify the data detection results based on different expected outcomes.

[0170] In a feasible data detection scenario, the aforementioned ABD in a abdIn the sequence, the data threshold within this block is 3, corresponding to table type t. a :order, the dcn node list is [AA0, AJ0, AI0]. C is in a c In the sequence, the data threshold within this block is 1, corresponding to table type t. a :order, the list of dcn nodes is [AK0].

[0171] Furthermore, based on the obtained DCN node configuration threshold of 4, it can be known that C is in a c In the sequence, the data threshold 1 is less than ABD in a abd The data threshold in the sequence is 3, indicating a data discrepancy at node C. Therefore, in the ABCD dataset, ABD can be considered to have the same characteristics at nodes AA0, AJ0, and AI0 in dcn, and no data discrepancy exists. However, C exhibits a data anomaly at node AK0, triggering event-mode processing.

[0172] This application embodiment uses a real-time data sentinel to detect the status of the dataset and simultaneously processes data sets from various different data sources. Data aggregation ensures the timeliness of streaming computation. By determining the dynamic window calculation range through dual time dimensions, data accuracy is improved. If the data sequence pairs calculated by the streaming window do not match expectations, an event mechanism will be triggered to notify, identifying online anomalies, reducing the impact on business operations, and improving system fault tolerance. This application employs streaming computation to analyze large-scale flowing data in real time during its constantly changing movement, aggregating potentially useful information, and sending the results to the next computation node.

[0173] The following continues to describe the exemplary structure of the data detection device 154 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 1 As shown, the software module stored in the data detection device 154 of the memory 150 can be the data detection device in the terminal 100, including:

[0174] Preprocessing module 1541 is used to query the data of the distributed application server nodes and format the queried data to obtain a data model object that conforms to the predefined data model.

[0175] The aggregation module 1542 is used to distribute data model objects to corresponding channels for aggregation, obtain aggregated data sequence pairs, and gather the various data model objects associated with the aggregated data sequence pairs into the data stream;

[0176] The window setting module 1543 is used to set the dynamic data window range based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream.

[0177] The detection result matching module 1544 is used to process aggregated data sequence pairs in the data stream based on the set dynamic data window range, and generate detection results for the queried data; wherein, the detection results include whether the queried data is consistent with the expected data.

[0178] In some embodiments of this application, the preprocessing module 1541 is used to obtain data sequence pairs of data model objects; wherein, the data sequence pairs include data entity feature sequences and data information feature sequences;

[0179] Preprocessing module 1541 is used to determine the queue corresponding to the feature values ​​contained in the feature sequence of the data entity;

[0180] The aggregation module 1542 is used to distribute data model objects to corresponding channels for aggregation in at least one defined queue, so as to obtain aggregated data sequence pairs.

[0181] In some embodiments of this application, the preprocessing module 1541 is used to obtain the data length, product coefficient, modulo coefficient, and encoding standard value of the string mapping contained in the data model object; determine the feature value of the string based on the product coefficient, modulo coefficient, data length, and encoding standard value of the string mapping, and use the feature values ​​of all strings contained in the data model object as the data entity feature sequence of the data model object; obtain the source database area, data source table, and data query time of the data model object, and use the source database area, data source table, and data query time as the data information feature sequence of the data model object; and form a data sequence pair of the data model object based on the data entity feature sequence and the data information feature sequence.

[0182] In some embodiments of this application, the same feature value corresponds to a queue. The aggregation module 1542 is used to create a new temporary metric block for the same data entity feature sequence, and to mark the temporary metric block with the aggregated time series and the same block color.

[0183] The aggregation module 1542 is used to extract query time series, source database region, and data source table from the data entity feature sequence of the data model object;

[0184] The aggregation module 1542 is used to obtain aggregated data sequence pairs based on feature values, source database regions, data source tables, query time series, block colors, and aggregation time series.

[0185] In some embodiments of this application, the start time is the data query time of each data model object, the end time is the aggregation time of each data model object, and the detection result matching module 1544 is used to obtain the set predetermined window and delay window; the time range covered by the minimum time node to the maximum time node contained in the data query time and aggregation time is used as the dynamic adjustment window; and the dynamic data window range is set based on the predetermined window, delay window and dynamic adjustment window.

[0186] In some embodiments of this application, the detection result matching module 1544 is used to retrieve the minimum and maximum time nodes in the data query time and aggregation time based on the sparse table.

[0187] In some embodiments of this application, the expected data is the configuration threshold of the distributed application server node. Based on the set dynamic data window range, the detection result matching module 1544 is used to perform operations on the aggregated data sequence pairs within the dynamic data window range to filter out each color block and determine the data threshold within each color block; in the aggregated data sequence within the dynamic data window range, the feature quantity value of each color block is determined; wherein, the feature quantity value represents the number of data with the same feature value; based on the data threshold within each color block, the feature quantity value of each color block, and the configuration threshold, a detection result is generated.

[0188] The data detection device provided in this application queries data from distributed application server nodes and formats the queried data to obtain data model objects that conform to a predefined data model. In other words, the data detection method provided in this application can process data from distributed application server nodes, i.e., data from different data sources, and format the data from different data sources, supporting cross-domain and cross-database data detection. Furthermore, the data model objects are distributed to corresponding channels for aggregation, resulting in aggregated data sequence pairs. The data model objects associated with each aggregated data sequence pair are then gathered into a data stream. Based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream... In this process, a dynamic data window range is set. Based on the set dynamic data window range, the aggregated data sequence pairs in the data stream are processed to generate detection results for the queried data. The detection results include whether the queried data is consistent with the expected data. Here, defining an appropriate data range based on the set dynamic data window range for detection not only ensures data integrity, but also, since streaming computation based on the dynamic data window range is oriented towards an infinite input data stream, it raises the threshold for data volume entry, can handle large amounts of data flow, and improves detection efficiency. Thus, the data detection method provided in this application achieves the goal of supporting cross-domain data consistency detection and improving detection efficiency in cross-domain and cross-database types.

[0189] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0190] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 2 The method shown.

[0191] The computer-readable storage medium provided in this application queries data from distributed application server nodes and formats the queried data to obtain data model objects conforming to a predefined data model. In other words, the data detection method provided in this application can process data from distributed application server nodes, i.e., data from different data sources, and format the data from different data sources, supporting cross-domain and cross-database data detection. Furthermore, the data model objects are distributed to corresponding channels for aggregation, resulting in aggregated data sequence pairs, and the data model objects associated with the aggregated data sequence pairs are gathered into a data stream. Based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream... The method involves setting a dynamic data window range and processing aggregated data sequence pairs in the data stream based on this range to generate detection results for the queried data. These results include whether the queried data matches the expected data. By defining a suitable data range within the dynamic data window range, the method not only ensures data integrity but also addresses the issue of infinite input data streams through streaming computation, thus raising the threshold for data volume acceptance and enabling the handling of large volumes of data, thereby improving detection efficiency. Therefore, the data detection method provided in this application achieves the goal of supporting cross-domain data consistency detection and improving detection efficiency in cross-domain and cross-database scenarios.

[0192] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0193] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0194] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may also be stored as part of a file containing other programs or data, for example, in one or more scripts within a HyperText Markup Language (HText Markup Language) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0195] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data detection method, characterized in that, The method includes: The system queries the data of the distributed application server nodes and formats the retrieved data to obtain a data model object that conforms to a predefined data model. The data model objects are distributed to the corresponding channels for aggregation to obtain aggregated data sequence pairs, and the data model objects associated with the aggregated data sequence pairs are gathered into the data stream; Based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream, a dynamic data window range is set; Based on the set dynamic data window range, the aggregated data sequence pairs in the data stream are processed to generate detection results for the queried data; wherein, the detection results include whether the queried data is consistent with the expected data; The start time is the data query time of each data model object, and the end time is the aggregation time of each data model object. The dynamic data window range is set based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream, including: Get the scheduled window and delay window settings; The time range covered by the minimum and maximum time nodes contained in the data query time and the aggregation time is used as a dynamic adjustment window. The range of the dynamic data window is set based on the predetermined window, the delay window, and the dynamic adjustment window.

2. The method according to claim 1, characterized in that, The step of splitting the data model object into corresponding channels for aggregation to obtain aggregated data sequence pairs includes: Obtain data sequence pairs of the data model object; wherein, the data sequence pairs include data entity feature sequences and data information feature sequences; Determine the queue corresponding to the feature values ​​contained in the feature sequence of the data entity; The data model object is distributed to the corresponding channel for aggregation in at least one defined queue to obtain the aggregated data sequence pair.

3. The method according to claim 2, characterized in that, Obtaining the data sequence pairs of the data model object includes: Obtain the data length, product coefficient, modulo coefficient, and encoding standard value of the string mapping contained in the data model object; Based on the product coefficient, the modulus coefficient, the data length, and the encoding standard value of the string mapping, the feature value of the string is determined, and the feature values ​​of all strings contained in the data model object are used as the data entity feature sequence of the data model object. Obtain the source database region, data source table, and data query time of the data model object, and use the source database region, data source table, and data query time as the data information feature sequence of the data model object; The data sequence pairs that form the data model object are based on the data entity feature sequence and the data information feature sequence.

4. The method according to claim 2, characterized in that, Each feature value corresponds to a queue. The data model object is distributed to corresponding channels for aggregation within at least one defined queue to obtain the aggregated data sequence pairs, including: Create a new temporary metric block for the same data entity feature sequence, and label the temporary metric block with the aggregated time series and the same block color; Extract the query time series, source database region, and data source table from the data entity feature sequence of the data model object; Based on the feature value, the source database region, the data source table, the query time series, the block color, and the aggregation time series, the aggregated data sequence pair is obtained.

5. The method according to claim 1, characterized in that, Before using the time range covered by the smallest to largest time node included in the data query time and the aggregation time as the dynamic adjustment window, the method further includes: Based on the sparse table, the minimum time node and the maximum time node are retrieved from the data query time and the aggregation time.

6. The method according to any one of claims 1 to 4, characterized in that, The expected data is the configuration threshold of the distributed application server node. The process of processing aggregated data sequence pairs in the data stream based on the set dynamic data window range to generate detection results for the queried data includes: The aggregated data sequence pairs within the dynamic data window are processed to filter out each color block and determine the data threshold within each color block; Within the aggregated data sequence within the dynamic data window, a feature quantity value is determined for each color block; wherein, the feature quantity value represents the number of data with the same feature value. The detection result is generated based on the data threshold within each color block, the feature quantity value of each color block, and the configuration threshold.

7. A data detection device, characterized in that, The device includes: The preprocessing module is used to query the data of the distributed application server nodes and format the queried data to obtain a data model object that conforms to the predefined data model. The aggregation module is used to distribute data model objects to corresponding channels for aggregation, obtain aggregated data sequence pairs, and gather the data model objects associated with the aggregated data sequence pairs into the data stream; The window setting module is used to set the dynamic data window range based on the start time of each data model object entering the data stream and the end time of each data model object in the data stream; The detection result matching module is used to process the aggregated data sequence pairs in the data stream based on a set dynamic data window range, and generate detection results for the queried data; wherein, the detection results include whether the queried data is consistent with the expected data; The start time is the data query time for each data model object, and the end time is the aggregation time for each data model object. The detection result matching module is used to obtain the set pre-defined window and delay window; the time range covered by the minimum and maximum time nodes contained in the data query time and aggregation time is used as the dynamic adjustment window; the dynamic data window range is set based on the pre-defined window, delay window, and dynamic adjustment window.

8. A data detection device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores executable instructions for causing a processor to execute, thereby implementing the method of any one of claims 1 to 6.

Images