Data Difference Monitoring Method, Apparatus, Electronic Device, and Computer Readable Medium

Abstract

Embodiments of the present disclosure disclose a data difference monitoring method, apparatus, electronic device, and computer-readable medium. A specific implementation of the method includes: performing node configuration on a data node information set to obtain a configured node information set; performing node information connection on the configured node information set to obtain a configured node link set; for each configured node link in the configured node link set, performing the following storage steps: performing monitoring information configuration on the configured node link to obtain a monitoring information set; extracting data from the configured node sequence according to the monitoring information set to obtain a temporary data table sequence; performing data verification on the temporary data table sequence to obtain a verification result; and in response to determining that there are data differences, storing the difference data. This implementation can expand the scope of data monitoring, improve the efficiency and accuracy of data verification, and thereby reasonably adjust the warehouse inventory by constructing a link to simultaneously perform real-time monitoring on multiple systems.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technologies, and more particularly, to methods, apparatuses, electronic devices, and computer-readable media for monitoring data differences. Background Art

[0002] With the continuous increase in systems, there will be data transfer between multiple systems. However, during the data transfer process, problems such as data loss and data corruption are likely to occur, resulting in data inconsistency between multiple systems. For data difference monitoring, the commonly used method is to monitor the data transfer process and perform data verification between two systems.

[0003] However, the inventors have found that when using the above method to monitor data differences, the following technical problems often exist:

[0004] First, only monitoring and verifying the differential data between two systems results in a relatively small monitoring scope, leading to a long monitoring time and low monitoring efficiency, and further causing waste of warehouse inventory resources or item accumulation.

[0005] Second, due to the large volume of data transfer data, directly verifying the data between systems easily causes a large system load, resulting in a low accuracy rate of data verification, and further causing waste of warehouse inventory resources or item accumulation.

[0006] The above information disclosed in this background art section is only used to enhance the understanding of the background of the inventive concept, and thus, it may include information that does not form the prior art known to those of ordinary skill in the art in this country. Summary of the Invention

[0007] The content part of the present disclosure is used to briefly introduce concepts, which will be described in detail in the following detailed implementation part. The content part of the present disclosure is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0008] Some embodiments of the present disclosure propose methods, apparatuses, electronic devices, and computer-readable media for monitoring data differences to solve one or more of the technical problems mentioned in the above background art section.

[0009] In a first aspect, some embodiments of the present disclosure provide a data difference monitoring method, including: performing node configuration on each data node corresponding to the data node information in the data node information set to generate configured node information and obtain a configured node information set; connecting the node information in the above-mentioned configured node information set according to the order of data flow to obtain a configured node link set, where the order of the above-mentioned data flow is the order in which data is transmitted in the data nodes; for each configured node link in the above-mentioned configured node link set, performing the following storage steps: configuring monitoring information for the above-mentioned configured node link to obtain a monitoring information set; sequentially extracting data from the configured node sequence corresponding to the above-mentioned configured node link according to the above-mentioned monitoring information set to obtain a temporary data table sequence corresponding to the above-mentioned configured node sequence, where there is a one-to-one correspondence in quantity between the above-mentioned configured node sequence and the above-mentioned temporary data table sequence; performing data verification on the above-mentioned temporary data table sequence to obtain a verification result; in response to determining that there are differences in the data in the above-mentioned temporary data table sequence in the above-mentioned verification result, storing the difference data in the above-mentioned temporary data table sequence.

[0010] In a second aspect, some embodiments of the present disclosure provide a data difference monitoring device, including: a node configuration unit configured to perform node configuration on each data node corresponding to the data node information in the data node information set to generate configured node information and obtain a configured node information set; a node information connection unit configured to connect the node information in the above-mentioned configured node information set according to the order of data flow to obtain a configured node link set, where the order of the above-mentioned data flow is the order in which data is transmitted in the data nodes; an execution unit configured to, for each configured node link in the above-mentioned configured node link set, perform the following storage steps: configuring monitoring information for the above-mentioned configured node link to obtain a monitoring information set; sequentially extracting data from the configured node sequence corresponding to the above-mentioned configured node link according to the above-mentioned monitoring information set to obtain a temporary data table sequence corresponding to the above-mentioned configured node sequence, where there is a one-to-one correspondence in quantity between the above-mentioned configured node sequence and the above-mentioned temporary data table sequence; performing data verification on the above-mentioned temporary data table sequence to obtain a verification result; in response to determining that there are differences in the data in the above-mentioned temporary data table sequence in the above-mentioned verification result, storing the difference data in the above-mentioned temporary data table sequence.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device, including: one or more processors; a storage device storing one or more programs thereon, and when the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the method described in any implementation manner in the first aspect.

[0012] Fourthly, some embodiments of the present disclosure provide a computer-readable medium, on which a computer program is stored, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0013] The above various embodiments of the present disclosure have the following beneficial effects: The data difference monitoring method of some embodiments of the present disclosure can expand the scope of data monitoring, improve the data verification efficiency and accuracy, and then reasonably adjust the warehouse inventory by constructing a link to simultaneously monitor multiple systems in real time. Specifically, the reasons for the waste of relevant warehouse inventory resources or the accumulation of items are as follows: Only monitoring and verifying the difference data between two systems results in a small monitoring scope, long monitoring time, and low monitoring efficiency, which in turn leads to the waste of warehouse inventory resources or the accumulation of items. Based on this, the data difference monitoring method of some embodiments of the present disclosure can first configure each data node corresponding to the data node information in the data node information set to generate configuration node information, obtaining a configuration node information set. Here, performing data node configuration can abstract the system into nodes to avoid high system load. Then, according to the order of data flow, connect the node information of the above configuration node information set to obtain a configuration node link set, where the above order of data flow is the order in which data is transmitted in the data node. Here, connecting the node information of the configuration node information set according to the order of data flow can avoid repeated data verification of data nodes and improve the data verification efficiency. Finally, for each configuration node link in the above configuration node link set, perform the following storage steps: Configure monitoring information for the above configuration node link to obtain a monitoring information set. Here, performing monitoring information configuration can monitor the configuration node in real time and improve the timeliness of data verification. According to the above monitoring information set, sequentially extract data from the configuration node sequence corresponding to the above configuration node link to obtain a temporary data table sequence corresponding to the above configuration node sequence, where there is a one-to-one correspondence in quantity between the above configuration node sequence and the above temporary data table sequence. Here, obtaining the temporary data table sequence by data extraction can be applicable to the data verification scenario of a large amount of data, and can avoid causing too high a system load on the configuration node and improve the data verification efficiency. Verify the data in the above temporary data table sequence to obtain a verification result. In response to determining that the verification result is that there are differences in the data in the above temporary data table sequence, store the difference data in the above temporary data table sequence. Here, storing the difference data can facilitate the maintenance personnel to view and improve the processing efficiency of the difference data. Thus, it can be seen that the data difference monitoring method can expand the scope of data monitoring, improve the data verification efficiency and accuracy, and then reasonably adjust the warehouse inventory by constructing a link to simultaneously monitor multiple systems in real time. Description of the Drawings

[0014] In conjunction with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the accompanying drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the elements and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flowchart of some embodiments of a data difference monitoring method according to the present disclosure;

[0016] Figure 2 is a schematic structural diagram of some embodiments of a data difference monitoring device according to the present disclosure;

[0017] Figure 3 is a schematic structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. Specific Embodiments

[0018] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the accompanying drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0019] In addition, it should be noted that for the sake of convenience of description, only the parts related to the relevant invention are shown in the accompanying drawings. Without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules, or units, and are not used to limit the order or mutual dependency relationship of the functions performed by these devices, modules, or units.

[0021] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly stated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0024] Reference Figure 1, which shows the flow 100 of some embodiments of the data difference monitoring method according to the present disclosure. The data difference monitoring method includes the following steps:

[0025] Step 101, perform node configuration on each data node corresponding to the data node information in the data node information set to generate configuration node information, and obtain a configuration node information set.

[0026] In some embodiments, the execution subject (e.g., an electronic device) of the above data difference monitoring method can perform node configuration on each data node corresponding to the data node information in the data node information set to generate configuration node information, and obtain a configuration node information set. Among them, the above data node can be a system for storing data. The above data node can also be a hardware device for storing data. The above data node information can be the data number information corresponding to the above data node. The above configuration node information can be the node information for performing node configuration on the data node. The above node configuration can include, but is not limited to, at least one of the following: data query function configuration, data transfer function configuration. The above data query function configuration can be the configuration of the data query function for the data node set corresponding to the data node information set. The above data transfer function configuration can be the configuration of data transmission between the data node sets corresponding to the data node information set.

[0027] In some alternative implementation manners of some embodiments, the above performing node configuration on each data node corresponding to the data node information in the data node information set to generate configuration node information, and obtaining a configuration node information set, may include the following steps:

[0028] The first step is to perform the following configuration steps for each data node information in the above data node information set:

[0029] Sub-step 1, perform index information configuration on the data node corresponding to the above data node information according to a preset index pool to obtain index information configuration data node information. Among them, the above preset index pool can be a pre-set index pool for storing index information related to data nodes. The above index information can be information characterizing the degree of development of things. For example, the above index information can be order quantity information. The above preset index pool can include, but is not limited to, at least one of the following: order quantity, item transfer quantity, and item value attribute. The above index information configuration data node information can be the data node information after configuring the index information.

[0030] As an example, the above execution subject can first screen out the information with the index information being order quantity information from the above preset index pool. Then, configure the above order quantity information to the data node corresponding to the above data node information to obtain index information configuration data node information.

[0031] Sub-step 2: According to the preset dimension pool, perform dimension configuration on the index information configuration data node corresponding to the data node information of the above index information to obtain dimension information configuration data node information. Among them, the above preset dimension pool can be a pre-configured dimension pool that stores dimension information related to the index information configuration data node. The above dimension information can be information characterizing the characteristic attributes of the data node. For example, the above dimension information can be venue number information. The above preset dimension pool can include but is not limited to at least one of the following: venue number and verification time range. The above dimension information configuration data node information can be the index information configuration data node information after configuring the dimension information.

[0032] As an example, the above execution entity can first screen out the information with dimension information being venue number information and verification time range information from the above preset dimension pool. Then, configure the above venue number information and verification time range information to the index information configuration data node corresponding to the above index information configuration data node information to obtain dimension information configuration data node information.

[0033] Sub-step 3: Perform data source configuration on the dimension information configuration data node corresponding to the above dimension information configuration data node information to obtain data source configuration data node information. Among them, the above data source configuration data node information can be the node information for storing the data to be verified in the data source. The above data source can be a data source that supports real-time data storage. For example, the above data source can be StarRocks.

[0034] Sub-step 4: Perform data collection information configuration on the data source configuration data node corresponding to the above data source configuration data node information to obtain configuration node information. Among them, the above data collection information can be the data information to be verified. The above configuration node information can be the node information after configuring the index information, dimension information, data source information, and collected data information for the data node.

[0035] Step 102: According to the order of data flow, connect the node information in the configuration node information set to obtain a configuration node link set.

[0036] In some embodiments, the above execution entity can connect the node information in the above configuration node information set according to the order of data flow to obtain a configuration node link set. Among them, the above order of data flow is the order in which data is transmitted in the data node. The configuration node link in the above configuration node link set can be a link formed by connecting the configuration nodes passed by the data flow in a directed edge manner according to the order of data flow.

[0037] As an example, the above execution entity may construct directed edges for the set of configuration nodes passed through by the above data flow according to the order of data flow, to obtain a set of configuration node linkages.

[0038] Step 103: For each configuration node linkage in the set of configuration node linkages, perform the following storage steps:

[0039] Step 1031: Configure monitoring information for the configuration node linkage to obtain a set of monitoring information.

[0040] In some embodiments, the above execution entity may configure monitoring information for the above configuration node linkage to obtain a set of monitoring information. Among them, the monitoring information in the above set of monitoring information may include at least one of the following: monitoring metric information, monitoring dimension information. In practice, the above execution entity may add monitoring information to the above configuration node linkage to obtain a set of monitoring information.

[0041] In some optional implementation manners of some embodiments, the above configuring monitoring information for the above configuration node linkage to obtain a set of monitoring information may include the following steps:

[0042] First step: Configure monitoring metric information for the above configuration node linkage to obtain monitoring metric information as the first monitoring information. The above monitoring metric information may be the metric information that the configuration node linkage needs to monitor. For example, the above monitoring metric information may be order quantity information.

[0043] Second step: Configure monitoring dimension information for the above configuration node linkage to obtain monitoring dimension information as the second monitoring information. The above monitoring dimension information may be the dimension information that the configuration node linkage needs to monitor. For example, the above monitoring dimension information may include: venue number information and monitoring time range information.

[0044] Third step: Determine the above first monitoring information and the above second monitoring information as a set of monitoring information.

[0045] Step 1032: According to the set of monitoring information, sequentially extract data from the configuration node sequence corresponding to the configuration node linkage to obtain a sequence of temporary data tables corresponding to the configuration node sequence.

[0046] In some embodiments, the above-mentioned execution entity may sequentially extract data from the configuration node sequence corresponding to the above-mentioned configuration node link according to the above-mentioned monitoring information set, obtain a sequence of temporary data tables corresponding to the above-mentioned configuration node sequence, and synchronize the obtained sequence of temporary data tables to the Spark platform. Among them, the above-mentioned configuration node sequence may be a sequence obtained by sorting the configuration node set corresponding to the configuration node link in the order of data flow. There is a one-to-one correspondence between the above-mentioned configuration node sequence and the above-mentioned sequence of temporary data tables. For example, the above-mentioned configuration node sequence includes: a first configuration node and a second configuration node. The above-mentioned sequence of temporary data tables includes: a first temporary data table and a second temporary data table. The above-mentioned first temporary data table is a data table obtained by extracting data from the above-mentioned first configuration node. The above-mentioned second temporary data table is a data table obtained by extracting data from the above-mentioned second configuration node. The temporary data tables in the above-mentioned sequence of temporary data tables may be data tables storing data extracted from configuration nodes. The above-mentioned data extraction may be to extract data from the data source corresponding to the above-mentioned configuration node set.

[0047] As an example, the above-mentioned execution entity may determine the monitoring metric information and monitoring dimension information that need to be monitored for the above-mentioned configuration node link through the first monitoring information and the second monitoring information in the monitoring information set. Then, combine the above-mentioned monitoring metric information and monitoring dimension information into a corresponding target query statement. Among them, the above-mentioned target query statement may be a query statement for extracting data from the above-mentioned configuration node. Finally, use the above-mentioned target query statement to extract data from the configuration node sequence corresponding to the above-mentioned configuration node link, and obtain a sequence of temporary data tables corresponding to the above-mentioned configuration node set.

[0048] In some optional implementation manners of some embodiments, the above-mentioned sequentially extracting data from the configuration node sequence corresponding to the above-mentioned configuration node link according to the above-mentioned monitoring information set to obtain a sequence of temporary data tables corresponding to the above-mentioned configuration node sequence may include the following steps:

[0049] First step, perform dynamic cursor configuration on the configuration node at the starting position in the above-mentioned configuration node link. Among them, a dynamic cursor may be a mechanism for extracting one record each time from a result set including multiple data records. A dynamic cursor may be a cursor for modifying each row of data in the temporary data table corresponding to the starting configuration node. The above-mentioned starting configuration node may be the configuration node at the starting position. In practice, the above-mentioned execution entity may add a dynamic cursor to the configuration node at the starting position in the above-mentioned configuration node link, so that the configuration node at the starting position can perform data verification one by one among multiple pieces of data to be verified, transfer data to the configuration node for data extraction, and avoid repeated data verification, and the dynamic cursor supports incremental data acquisition.

[0050] Step 2: Determine the association relationships between the metric information and dimension information of the configuration node sequence corresponding to the above configuration node link and the monitoring metric information and the above monitoring dimension information in the above monitoring information set. Among them, the above association relationships can be the corresponding relationships between attribute names. The attribute names can include: the attribute name of the metric information, the attribute name of the dimension information, the attribute name of the monitoring metric information, and the attribute name of the monitoring dimension information. The above association relationships can include: the corresponding relationship between the metric information of the above configuration node and the attribute name of the monitoring metric information in the monitoring information set, and the corresponding relationship between the dimension information of the above configuration node and the attribute name of the monitoring dimension information. For example, the above configuration node link includes: a first configuration node and a second configuration node. The above monitoring metric information can be order quantity information. The above monitoring dimension information can include: venue number information and monitoring time range information. The attribute name corresponding to the venue number information in the first configuration node can be store_no, and the attribute name corresponding to the order quantity information can be order_frequency. The attribute name corresponding to the venue number information in the second configuration node can be store_num, and the attribute name corresponding to the order quantity information can be order_count. Since the attribute names corresponding to the venue number information in the first configuration node and the venue number information in the second configuration node are inconsistent, and the attribute names corresponding to the order quantity information in the first configuration node and the order quantity information in the second configuration node are inconsistent, it is necessary to establish an association relationship between the venue number information in the first configuration node and the venue number information in the second configuration node, that is, set a venue number alias information when querying the venue number information in the first configuration node and the venue number information in the second configuration node. The venue number alias information can be the venue number store_id in the monitoring dimension information. Establish an association relationship between the order quantity information in the first configuration node and the order quantity information in the second configuration node, that is, set a unified order quantity alias information when querying the order quantity information in the first configuration node and the order quantity information in the second configuration node. The order quantity alias information can be the order quantity information order_cnt in the monitoring dimension information.

[0051] Step 3: According to the above association relationships, use a dynamic cursor to extract data for each configuration node corresponding to the configuration node information in the above configuration node information sequence to obtain a sequence of temporary data tables.

[0052] As an example, the above execution entity can first, through the above association relationships, determine the data that needs to be extracted for each configuration node corresponding to the configuration node information in the above configuration node information sequence. Then, store the extracted data in the sequence of temporary data tables.

[0053] Step 1033: Check the data in the sequence of temporary data tables to obtain a check result.

[0054] In some embodiments, the above-mentioned execution entity may perform data verification on the above-mentioned temporary data table sequence to obtain a verification result. The verification result includes: there are differences in the data in the temporary data table sequence, and there are no differences in the data in the temporary data table sequence. The above data verification may be data verification performed on the spark platform. The spark platform may be a platform for processing massive data, thereby avoiding excessive load on the configuration nodes, resulting in reduced data verification efficiency and low verification accuracy.

[0055] As an example, the above-mentioned execution entity may generate a data verification statement through the association relationship included in the monitoring information. Then, use the data verification statement to perform data verification on the data in the above-mentioned temporary data table sequence to obtain a verification result.

[0056] In some optional implementation manners of some embodiments, the above-mentioned performing data verification on the above-mentioned temporary data table sequence to obtain a verification result may include the following steps:

[0057] First step, according to the above-mentioned association rule, perform deduplication on the above-mentioned temporary data table sequence to obtain a deduplicated data table set.

[0058] As an example, the above-mentioned execution entity may first obtain the association rule to determine whether there is duplicate extraction of the configuration node sequence corresponding to the above-mentioned configuration node information sequence. Then, in response to determining that there is duplicate extraction, perform deduplication processing on the temporary data table sequence corresponding to the duplicate extraction to obtain a deduplicated data table sequence. Finally, in response to determining that there is no duplicate extraction, determine the temporary data table sequence as the deduplicated data table sequence.

[0059] Second step, split each deduplicated data table in the above-mentioned deduplicated data table sequence to obtain a split data table sequence. The above-mentioned split data table sequence may be a sequence obtained by splitting the above-mentioned deduplicated data table sequence. In practice, the above-mentioned execution entity may split each deduplicated data table in the above-mentioned deduplicated data table sequence according to the primary key or index in the temporary data table to obtain a split data table sequence.

[0060] Third step, perform information summary processing on the above-mentioned split data table sequence to obtain a processed data table sequence. In practice, the above-mentioned execution entity may use MD5 (Message-Digest Algorithm) to perform information summary processing on the above-mentioned split data table sequence to obtain a processed data table sequence.

[0061] Step 4: Send the processed data table sequence to the big data processing platform. Among them, the big data processing platform can be a platform for processing massive data. For example, the big data processing platform can be a Spark platform. The relevant processing can include at least one of the following: batch processing, real-time stream processing.

[0062] Step 5: Use the preset verification statement to verify the processed data table sequence on the big data processing platform to obtain a verification result. Among them, the preset verification statement can be a pre-generated verification statement obtained based on different association relationships. The preset verification rule can be obtained through the following steps:

[0063] Sub-step 1: Obtain the historical verification statement set. Among them, the historical verification statements in the historical verification statement set can be statements for verifying data before the current time.

[0064] Sub-step 2: Extract features from the historical verification statement set to obtain a feature vector. Among them, the feature vector can be a vector representing the attributes of the historical verification statement set.

[0065] Sub-step 3: Input the feature vector into the statement automatic generation model to obtain a preset verification statement. Among them, the statement automatic generation model can be a machine learning model or a neural network model. Machine learning models can include but are not limited to: mixture Gaussian model, hidden Markov model.

[0066] The above technical solution and its related content, as an inventive point of the embodiments of the present disclosure, solve the second technical problem mentioned in the background technology, "Due to the large amount of data flowing in the data transfer, directly performing data verification on the system is likely to cause a large system load, resulting in a low accuracy rate of data verification, and further leading to waste of warehouse inventory resources or accumulation of items." The factors leading to waste of warehouse inventory resources or accumulation of items are often as follows: Due to the large amount of data flowing in the data transfer, directly performing data verification on the system is likely to cause a large system load, resulting in a low accuracy rate of data verification, and further leading to waste of warehouse inventory resources or accumulation of items. To achieve this effect, the present disclosure first de-duplicates the above temporary data table sequence according to the above association rules to obtain a de-duplicated data table set. Here, the de-duplication process can avoid repeated verification of the temporary data table, reduce the amount of calculation, and improve the verification efficiency. Secondly, each de-duplicated data table in the above de-duplicated data table sequence is split to obtain a split data table sequence. Here, the splitting process can reduce the amount of data to be verified, reduce the system load, and improve the verification efficiency. Thirdly, information summarization processing is performed on the above split data table sequence to obtain a processed data table sequence. Here, performing information summarization processing can ensure data security and avoid data leakage. Then, the above processed data table set is sent to the big data processing platform. Here, sending it to the big data processing platform can avoid causing an excessive system load and can improve the data verification efficiency. Finally, using a preset verification statement, data verification is performed on the above processed data table sequence on the above big data processing platform to obtain a verification result. Among them, the above preset verification rule can be obtained through the following steps: obtaining a set of historical verification statements; performing feature extraction on the above set of historical verification statements to obtain feature vectors; inputting the above feature vectors into a statement automatic generation model to obtain a preset verification statement. Here, verifying data through the automatically generated verification statement can improve the data verification efficiency and reduce the labor cost. Thus, by splitting the temporary data table to be verified, sending it to the big data platform, and automatically generating verification statements, the system load can be reduced, the data verification efficiency can be improved, and the warehouse storage can be reasonably guided.

[0067] Step 1034, in response to determining that there are differences in the data in the temporary data table sequence, store the different data in the temporary data table sequence.

[0068] In some embodiments, the above execution subject may, in response to determining that there are differences in the data in the above temporary data table sequence, store the different data in the above temporary data table sequence.

[0069] Optionally, after the above step of, in response to determining that there are differences in the data in the above temporary data table sequence, storing the different data in the above temporary data table sequence, the following steps may further be included:

[0070] In response to determining that there are differences in the data in the above temporary data table sequence, an alarm message is triggered and the above alarm message is sent to the monitoring interface. Among them, the above alarm message is the information obtained by configuring the alarm message for the above configuration node link. The above monitoring interface can be the interface through which the above maintenance personnel can view the verification result in real time. The above alarm message can inform the information about the data differences on the configuration node link. The above alarm message includes: alarm person-in-charge information and alarm method information. The above alarm person-in-charge information can be the information of the person responsible for monitoring the configuration node link. For example, the above alarm method information can include: email alarm information, SMS alarm information, and phone call alarm information.

[0071] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: The data difference monitoring method of some embodiments of the present disclosure can expand the scope of data monitoring, improve the data verification efficiency and accuracy, and then reasonably adjust the warehouse inventory by constructing a link to simultaneously monitor multiple systems in real time. Specifically, the reasons for the waste of relevant warehouse inventory resources or the accumulation of items are as follows: Only monitoring and verifying the difference data between two systems results in a small monitoring scope, leading to a long monitoring time and low monitoring efficiency, and further causing the waste of warehouse inventory resources or the accumulation of items. Based on this, the data difference monitoring method of some embodiments of the present disclosure can first perform node configuration on the data nodes corresponding to each data node information in the data node information set to generate configured node information and obtain a configured node information set. Here, performing data node configuration can abstract the system into nodes to avoid high system load. Then, according to the order of data flow, connect the node information of the above-mentioned configured node information set to obtain a configured node link set, where the above-mentioned order of data flow is the order in which data is transmitted in the data nodes. Here, connecting the node information of the configured node information set according to the order of data flow can avoid duplicate data verification of data nodes and improve the data verification efficiency. Finally, for each configured node link in the above-mentioned configured node link set, perform the following storage steps: Configure monitoring information for the above-mentioned configured node link to obtain a monitoring information set. Here, performing monitoring information configuration can monitor the configured nodes in real time and improve the timeliness of data verification. According to the above-mentioned monitoring information set, sequentially extract data from the configured node sequence corresponding to the above-mentioned configured node link to obtain a temporary data table sequence corresponding to the above-mentioned configured node sequence, where there is a one-to-one correspondence in quantity between the above-mentioned configured node sequence and the above-mentioned temporary data table sequence. Here, obtaining the temporary data table sequence through data extraction can be applicable to the data verification scenario with a large amount of data, and can avoid causing too high a system load on the configured nodes and improve the data verification efficiency. Verify the data of the above-mentioned temporary data table sequence to obtain a verification result. In response to determining that the verification result is that there are differences in the data in the above-mentioned temporary data table sequence, store the difference data in the above-mentioned temporary data table sequence. Here, storing the difference data can facilitate the maintenance personnel to view and improve the processing efficiency of the difference data. Thus, it can be seen that the data difference monitoring method can expand the scope of data monitoring, improve the data verification efficiency and accuracy, and then reasonably adjust the warehouse inventory by constructing a link to simultaneously monitor multiple systems in real time.

[0072] Further referring to Figure 2 , as an implementation of the methods shown in the above figures, some embodiments of the present disclosure provide a data difference monitoring device, and these device embodiments are related to Figure 1Corresponding to the method embodiments shown, the data difference monitoring device can be specifically applied to various electronic devices.

[0073] As Figure 2 shown, a data difference monitoring device 200 includes: a node configuration unit 201, a node information connection unit 202, and an execution unit 203. Among them, the node configuration unit 201 is configured to: perform node configuration on each data node corresponding to the data node information in the data node information set to generate configured node information, and obtain a configured node information set. The node information connection unit 202 is configured to: connect the node information in the above configured node information set according to the order of data flow to obtain a configured node link set, where the above order of data flow is the order in which data is transmitted in the data nodes. The execution unit 203 is configured to: for each configured node link in the above configured node link set, perform the following storage steps: perform monitoring information configuration on the above configured node link to obtain a monitoring information set; according to the above monitoring information set, sequentially extract data from the configured node sequence corresponding to the above configured node link to obtain a temporary data table sequence corresponding to the above configured node sequence, where there is a one-to-one correspondence in quantity between the above configured node sequence and the above temporary data table sequence; perform data verification on the above temporary data table sequence to obtain a verification result; in response to determining that the data in the above temporary data table sequence is different, store the different data in the above temporary data table sequence.

[0074] It can be understood that the various units described in the data difference monitoring device 200 correspond to the respective steps in the method described with reference to Figure 1 Therefore, the operations, features, and beneficial effects described above for the method also apply to the data difference monitoring device 200 and the units included therein, and will not be elaborated here.

[0075] Next, refer to Figure 3 , which shows a schematic structural diagram of an electronic device (for example, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0076] As Figure 3As shown, the electronic device 300 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 301, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0077] Generally, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 3 an electronic device 300 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. More or fewer devices may be implemented or had alternatively. Figure 3 Each block shown in may represent one device or, as needed, multiple devices.

[0078] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for performing the methods shown in the flowcharts. In such some embodiments, the computer program may be downloaded and installed from a network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above functions defined in the methods of some embodiments of the present disclosure are executed.

[0079] It should be noted that in some embodiments of the present disclosure, the above-mentioned computer-readable medium may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0080] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed network.

[0081] The above computer-readable medium may be included in the above electronic device; or may exist separately without being assembled into the electronic device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device is caused to: perform node configuration on each data node corresponding to the data node information in the data node information set to generate configuration node information and obtain a configuration node information set; connect the node information in the above configuration node information set according to the order of data flow to obtain a configuration node link set, where the above order of data flow is the order in which data is transmitted in the data nodes; for each configuration node link in the above configuration node link set, perform the following storage steps: configure monitoring information for the above configuration node link to obtain a monitoring information set; sequentially extract data from the configuration node sequence corresponding to the above configuration node link according to the above monitoring information set to obtain a temporary data table sequence corresponding to the above configuration node sequence, where there is a one-to-one correspondence in quantity between the above configuration node sequence and the above temporary data table sequence; check the data in the above temporary data table sequence to obtain a check result; in response to determining that there are differences in the data in the above temporary data table sequence in the above check result, store the difference data in the above temporary data table sequence.

[0082] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or combinations thereof. The above programming languages include object-oriented programming languages - such as Java, Smalltalk, C++; and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or may be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that, in some alternative implementations, the functions denoted in the blocks may occur in a different order than that denoted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0084] The units described in some embodiments of the present disclosure may be implemented in software or in hardware. The described units may also be provided in a processor. For example, it may be described as: a processor includes a node configuration unit, a node information connection unit, and an execution unit. Among them, the names of these units do not constitute a limitation on the unit itself in some cases. For example, the node configuration unit may also be described as "a unit that configures each data node corresponding to the data node information in the data node information set to generate configuration node information and obtain a configuration node information set".

[0085] The functions described above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0086] The above description is only some preferred embodiments of the present disclosure and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, technical solutions formed by mutually replacing the above features with technical features having similar functions (but not limited to) disclosed in the embodiments of the present disclosure.

Claims

1. A data difference monitoring method, comprising: For each data node information in the data node information set, perform the following configuration steps: According to a preset metric pool, perform metric information configuration on the data node corresponding to the data node information to obtain metric information configuration data node information; According to a preset dimension pool, perform dimension configuration on the metric information configuration data node corresponding to the metric information configuration data node information to obtain dimension information configuration data node information; Perform data source configuration on the dimension information configuration data node corresponding to the dimension information configuration data node information to obtain data source configuration data node information; Perform data collection information configuration on the data source configuration data node corresponding to the data source configuration data node information to obtain configuration node information; According to the order of data flow, connect the node information in the configuration node information set to obtain a configuration node link set, where the order of data flow is the order in which data is transmitted in the data node; For each configuration node link in the configuration node link set, perform the following storage steps: Perform monitoring information configuration on the configuration node link to obtain a monitoring information set; According to the monitoring information set, sequentially extract data from the configuration node sequence corresponding to the configuration node link to obtain a temporary data table sequence corresponding to the configuration node sequence, where there is a one-to-one correspondence in quantity between the configuration node sequence and the temporary data table sequence; Perform data verification on the temporary data table sequence to obtain a verification result; In response to determining that there are differences in the data in the temporary data table sequence in the verification result, store the difference data in the temporary data table sequence.

2. The method according to claim 1, wherein After the step of storing the difference data in the temporary data table sequence in response to determining that there are differences in the data in the temporary data table sequence in the verification result, further comprising: In response to determining that there are differences in the data in the temporary data table sequence in the verification result, trigger an alarm message and send the alarm message to a monitoring interface, where the alarm message is information obtained by performing alarm information configuration on the configuration node link.

3. The method according to claim 1, wherein The step of performing monitoring information configuration on the configuration node link to obtain a monitoring information set includes: Perform monitoring metric information configuration on the configuration node link to obtain monitoring metric information as the first monitoring information; Perform monitoring dimension information configuration on the configuration node link to obtain monitoring dimension information as the second monitoring information; Determine the first monitoring information and the second monitoring information as the monitoring information set.

4. The method according to claim 3, wherein The step of sequentially extracting data from the configuration node sequence corresponding to the configuration node link according to the monitoring information set to obtain a temporary data table sequence corresponding to the configuration node sequence includes: Perform dynamic cursor configuration on the configuration node at the starting position in the configuration node link; Determine the association relationship between the metric information and dimension information of the configuration node sequence corresponding to the configuration node link and the monitoring metric information and monitoring dimension information in the monitoring information set; According to the association relationship, a dynamic cursor is used to extract data from the configuration node corresponding to each configuration node information in the configuration node information sequence, and a temporary data table sequence is obtained.

5. A data difference monitoring device, comprising: A node configuration unit, configured to perform the following configuration steps for each data node information in the data node information set: according to a preset index pool, perform index information configuration on the data node corresponding to the data node information to obtain index information configuration data node information; According to a preset dimension pool, perform dimension configuration on the index information configuration data node corresponding to the index information configuration data node information to obtain dimension information configuration data node information; Perform data source configuration on the dimension information configuration data node corresponding to the dimension information configuration data node information to obtain data source configuration data node information; Perform data collection information configuration on the data source configuration data node corresponding to the data source configuration data node information to obtain configuration node information; A node information connection unit, configured to connect node information in the configuration node information set according to the data flow sequence to obtain a configuration node link set, where the data flow sequence is the sequence in which data is transmitted in the data node; An execution unit, configured to perform the following storage steps for each configuration node link in the configuration node link set: perform monitoring information configuration on the configuration node link to obtain a monitoring information set; according to the monitoring information set, sequentially extract data from the configuration node sequence corresponding to the configuration node link to obtain a temporary data table sequence corresponding to the configuration node sequence, where there is a one-to-one correspondence in quantity between the configuration node sequence and the temporary data table sequence; perform data verification on the temporary data table sequence to obtain a verification result; in response to determining that there are differences in the data in the temporary data table sequence, store the difference data in the temporary data table sequence.

6. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-4.

7. A computer-readable medium having a computer program stored thereon, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-4.

Images