Data detection method and apparatus, device, and storage medium

Abstract

The present disclosure provides a data detection method, device and equipment and a storage medium, relates to the technical field of computers, in particular to the technical field of data processing. The specific technical scheme comprises the following steps: obtaining record information of target data in a target system; the record information comprises first record information corresponding to a starting node of the target data in the target system and / or second record information corresponding to a terminal node of the target data in the target system; determining a data state of the target data based on the record information; and determining effective state information of the target data in the target system based on the data state. The technical scheme of the present disclosure can reduce the invasiveness to the target system, while achieving detection of a single piece of data and improving the detection accuracy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular, to the technical field of data processing, and more particularly to a data detection method and device, equipment and storage medium. BACKGROUND

[0002] A content validation system such as a search engine or a recommendation engine has high requirements for the reliability and timeliness of data validation. It is particularly important to build an observation system for observing the data flow (or data validation flow) of the content validation system. The existing data flow observation system has high invasiveness to the content validation system, high access cost, and insufficient observation granularity, which is difficult to meet the actual needs. SUMMARY

[0003] The present disclosure provides a data detection method, device, equipment and storage medium.

[0004] According to a first aspect of the present disclosure, a data detection method is provided, comprising:

[0005] obtaining record information of target data in a target system; the record information comprises first record information corresponding to a starting node of the target data in the target system and / or second record information corresponding to a terminating node of the target data in the target system;

[0006] determining a data state of the target data based on the record information;

[0007] determining validation state information of the target data in the target system based on the data state.

[0008] According to a second aspect of the present disclosure, a data detection device is provided, comprising:

[0009] a first information obtaining module, configured to obtain record information of target data in a target system; the record information comprises first record information corresponding to a starting node of the target data in the target system and / or second record information corresponding to a terminating node of the target data in the target system;

[0010] a data state determining module, configured to determine a data state of the target data based on the record information;

[0011] a first validation state determining module, configured to determine validation state information of the target data in the target system based on the data state.

[0012] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data detection method provided by any embodiment of the present disclosure.

[0013] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the data detection method provided by any of the embodiments of the present disclosure.

[0014] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the data detection method provided by any of the embodiments of the present disclosure.

[0015] The technical solutions of the present disclosure can achieve at least the following beneficial effects:

[0016] Based on the record information of the target data in the target system, the data state of the target data is determined, the whole process can be processed asynchronously to reduce the influence on the performance of the target system, reduce the invasiveness to the target system, and at the same time, the detection of a single data can be realized, and the fineness of the detection is improved.

[0017] It should be understood that the contents described in this part are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings are used to better understand the present solution and do not constitute a limitation on the present disclosure. Among them:

[0019] Figure 1 is a structural framework of a content validation system and a principle diagram of a data detection device obtaining information of the content validation system according to an embodiment of the present disclosure;

[0020] Figure 2 is a flowchart of a data detection method according to an embodiment of the present disclosure;

[0021] Figure 3 is a partial flowchart of another data detection method according to an embodiment of the present disclosure;

[0022] Figure 4 is a principle diagram of a data detection method according to an embodiment of the present disclosure;

[0023] Figure 5 is a partial flowchart of another data detection method according to an embodiment of the present disclosure;

[0024] Figure 6 is a schematic diagram of the topology relationship between processing nodes being reversed in an embodiment of the present disclosure;

[0025] Figure 7 is a principle diagram of calculating a data aging index according to an embodiment of the present disclosure;

[0026] Figure 8 is a structural framework schematic diagram of a data detection device provided by an embodiment of the present disclosure;

[0027] Figure 9 is a structural framework schematic diagram of a data detection system provided by an embodiment of the present disclosure;

[0028] Figure 10 is a structural framework schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, which should be considered in a descriptive sense only and not limiting. Therefore, those of ordinary skill in the art should realize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0030] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0031] In the description of the embodiments of the present disclosure, it should be understood that the terms "first", "second" are used only for the purpose of description and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0032] It should be further understood that the phrase "and / or" used in the embodiments of the present disclosure includes all or any one of the associated listed items and all combinations thereof.

[0033] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used in the embodiments of the present disclosure have the same meaning as generally understood by those skilled in the art to which the present disclosure belongs.

[0034] First, the structural framework of the content activation system (or data activation system, activation system) disclosed herein is introduced. A content activation system typically includes multiple processing modules that process data from the input system. Adjacent processing modules can be connected via data transmission storage structures, such as message queues, more specifically, Apache Kafka message queues, or other methods. Its structural framework is as follows: Figure 1 The portion shown in the dashed box is as follows. Figure 1 Modules a, b, c, d, e, and f are all processing modules, and the connecting arrows between them represent message queues. Each processing module can contain multiple processing instances that process the data input to that module. Adjacent processing instances are typically connected by a storage structure that enables data transfer. The type of this storage structure can be the same as or different from the type of the storage structure between processing modules.

[0035] The inventors of this disclosure discovered that with the rapid growth of content business demands, the requirements for the timeliness and reliability of data activation in content activation systems are becoming increasingly stringent. On the one hand, most processing modules in content activation systems communicate asynchronously via message queues, necessitating real-time observation of the reliability of content activation. On the other hand, most content activation systems support users writing custom offline processing operators and their topology relationships, requiring real-time analysis of these custom topologies. At this point, traditional observation schemes are insufficient to fully observe the data flow. Specifically, traditional observation schemes mainly include the following:

[0036] First, data flow is traced by recording the span information of the call chain, using modules that support the OpenTracing protocol (an open distributed tracing specification), such as Skywalking (a distributed tracing system), for observation. Second, data metadata is written to a log storage system, such as Elasticsearch (a search server), through message queues between secondary consumption modules for analysis and querying. Third, parameters such as queries per second (QPS), latency, and message queue backlog are collected between processing modules to indirectly observe the health of the data flow.

[0037] The above observation scheme has the following drawbacks:

[0038] The scheme one has poor support effect on the content effective system. Specifically, the observation system of the scheme one has high invasiveness to the content effective system and has large access cost. The scheme one cannot realize observation on the processing of the data by the processing module to locate the bottleneck module, and can only observe when the data is output to the downstream message queue. For the case that the data is not output to the downstream message queue due to various abnormalities, the user can only find the abnormality and locate the bottleneck module when the user voluntarily queries. In the scene with huge data flow, in order to save log storage resources, most data flow tracking adopts the appending writing instead of the in-place updating, which further increases the observation difficulty. The scheme three can only observe the data flow at the module level, and cannot be refined to a single data. Meanwhile, the observation indexes such as the query rate per second, the delay and the message queue accumulation used by the scheme three cannot accurately find abnormal data flow in some special but important scenes. For example, in the data effective scene with small data flow but high timeliness requirement, the accumulation degree of the message queue between the processing modules cannot accurately reflect the health degree (reliability, timeliness, etc.) of the data flow.

[0039] The technical solutions of the present disclosure and how the technical solutions of the present disclosure solve the above technical problems will be described in detail below with specific embodiments.

[0040] According to the embodiments of the present disclosure, the present disclosure provides a data detection method, which can be applied to a data detection device, such as Figure 2 As shown in the figure, the method comprises:

[0041] S201, obtaining record information of target data in a target system.

[0042] The record information comprises first record information corresponding to a starting node of the target data in the target system and / or second record information corresponding to a terminal node of the target data in the target system.

[0043] S202, determining a data state of the target data based on the record information.

[0044] S203, determining effective state information of the target data in the target system based on the data state.

[0045] The data detection method provided by the present disclosure can determine the data state of the target data based on the record information of the target data in the target system. The whole process can be processed asynchronously to reduce the influence on the performance of the target system, reduce the invasiveness to the target system, and realize the detection of a single data to improve the detection accuracy.

[0046] The target system can be a content effective system as shown in Figure 1 The starting node can be the first processing module in the content effective system as shown in Figure 1The module a) or the first processing instance shown, the termination module can be the last processing module in the content validation system (for example Figure 1 The module f) or the last processing instance shown, the first record information corresponding to the starting node can be a storage structure between the first processing module and the second processing module or a storage structure between the first processing instance and the second processing instance, and the second record information corresponding to the termination node can be a storage structure between the second last processing module and the last processing module or a storage structure between the second last processing instance and the last processing instance. The storage structure can be a message queue, for example, an apache kafka message queue, but is not limited thereto. Referring to Figure 1 The example of

[0047] Referring to Figure 1 The example of Figure 1 When the data in the data stream is stored in the starting module corresponding to (for example Figure 3 When the target data is transmitted to the termination module corresponding to (for example

[0048] The record information can include at least one meta information (meta information) of the target data, such as identification information (logid or Id), time information (for example, processing time of the target data in the target system), tag information (tag, also referred to as label information), etc. The space occupied by the meta information is 2 to 4 orders of magnitude smaller than the space occupied by the complete data packet, which can greatly save the occupation of resources. In an actual test, the space occupied by the meta information is one thousandth of the space occupied by the complete data packet.

[0049] In the embodiments of the present disclosure, the target data can be module-level data or a single piece of data, so as to realize detection of different fine-grained data.

[0050] In an optional implementation, as shown in Figure 4 In the above step S202, the data state of the target data is determined based on the record information, including the following steps S301 to S303:

[0051] S301, based on the identification information of the target data, a target operation unit is allocated for the target data.

[0052] Optionally, as shown in Figure 5As shown, when the target data is allocated to the target operation unit, the distributed storage structure can be implemented, for example, through a distributed message queue, and other storage structures can also be implemented, and the embodiments of the present disclosure are not limited thereto, and the function of the distributed storage structure in the embodiments of the present disclosure can be implemented.

[0053] In one example, the record information corresponding to each data stream in the content validation system is stored in a distributed message queue, and the record information corresponding to each data stream is stored in a partition in the distributed queue. Each partition can transmit data in a FIFO (First Input First Output) manner, and one or a specified number of partitions can be allocated to a target operation unit, and then the record information in each partition is distributed to the corresponding target operation unit, so that the target operation unit processes the record information in one partition or a specified number of partitions. One target operation unit can be one processing thread.

[0054] In one example, the target operation unit can automatically obtain the record information in one or more than two partitions in the distributed storage structure through an automatic subscription mechanism, and then implement data consumption, state query and other processing of the specified partition.

[0055] In the above manner, the record information of the same data stream can be processed by the same target operation unit, avoiding error processing caused by cross processing of multiple data streams, and improving the accuracy of processing. In addition, the distributed storage structure has high scalability, and the corresponding target operation unit is allocated according to the partition and data transmission and processing, which can improve the scalability of the target operation unit.

[0056] S302, based on the identification information, store the record information in the first storage structure of the target operation unit.

[0057] Based on the identification information, the storage of the record information can improve the convenience of subsequent query, and the corresponding record information can be queried based on the identification information.

[0058] Optionally, the target operation unit can include a first storage structure, such as a map structure (a collection of key-value pairs), when distributing the record information to the target operation unit through the distributed message queue, for the target data in the same data stream, the identification information (which can be included in the record information) of the target data can be stored as a key, and the record information in the same partition can be stored as a value, to form a key-value pair stored in the map structure. The storage mode of the key-value pair can improve the subsequent query speed and support high-concurrency use scenarios.

[0059] S303, determining, by the target operation unit, the data state of the target data based on the record information stored in the first storage structure.

[0060] The record information can reflect the data state of the data corresponding to the start module and the end node of the same data stream, and further reflect the state of the whole data stream.

[0061] In another optional embodiment, as shown in Figure 5 Determining the data state of the target data based on the record information includes the following steps S501-S503.

[0062] S501, allocating a target operation unit for the target data based on the identification information of the target data.

[0063] The specific implementation of step S501 can refer to the related content of step S301 described above, and will not be repeated here.

[0064] S502, storing the record information in the first storage structure of the target operation unit based on the identification information.

[0065] The specific implementation of step S502 can refer to the related content of step S302 described above, and will not be repeated here.

[0066] S503, obtaining attribute information of the target data in the target system.

[0067] The record information can include at least one meta information (meta information) such as identification information (logid or Id) of the target data, time information, and tag information (tag).

[0068] S504, storing the attribute information in the second storage structure of the target operation unit based on the time information in the attribute information.

[0069] Optionally, the target operation unit can further include a second storage structure, which can be a message queue, for example, an apache kafka message queue, each element in the queue can save the attribute information of a target data, and can also be other storage structures, and the embodiments of the present disclosure are not limited thereto, and the function of the third data storage unit in the embodiments of the present disclosure can be realized.

[0070] In one example, the attribute information is stored in a distributed storage structure (for example, a distributed message queue with multiple shards), and is distributed to the second storage structure through the distributed storage structure. Specifically, for each target data in the same data stream, the storage time sequence of the attribute information corresponding to each target data can be determined based on the time information in the attribute information corresponding to each target data, and then the attribute information stored in the same shard is distributed to the second storage structure according to the storage time sequence.

[0071] The steps S503 and S504 can be executed synchronously with the steps S501 and S502 (refer to Figure 4 ), or can be executed before or after the steps S501 and S502, or can be executed in other orders according to actual needs, which are not limited in the embodiments of the present disclosure.

[0072] S505, reading the attribute information from the second storage structure based on the reading timing of the second storage structure by using the target operation unit.

[0073] The reading timing of the second storage structure is consistent with the storage timing, and the attribute information stored in the second storage structure first can be read first.

[0074] S506, reading the corresponding record information from the first storage structure based on the read attribute information by using the target operation unit.

[0075] In the case that the attribute information and the record information both include the positioning identification information, the corresponding record information of the same target data in the first storage structure can be quickly read based on the attribute information read from the second storage structure.

[0076] The pointer representing the reference relationship between the positioning identification information and the element in the second storage structure can also be stored in the first storage structure, so as to facilitate the reference of the attribute information stored in the element in the second storage structure when needed.

[0077] S507, determining the data state of the target data based on the record information.

[0078] In an optional embodiment, the determination of the data state of the target data based on the record information includes at least one of the following four ways:

[0079] In the case that the first record information corresponding to the starting node and the second record information corresponding to the terminal node are both collected based on the record information, the determination of the data state includes that the data stream corresponding to the target data is a normal data stream.

[0080] In this way, the first record information corresponding to the starting node and the second record information corresponding to the terminal node are both collected, which are complete record information from the beginning to the end, and thus it is determined that the corresponding data stream is normally completed.

[0081] In the case that the second record information is collected and the first record information is not collected based on the record information, the determination of the data state includes that the data stream corresponding to the target data is a normal data stream.

[0082] In this mode, since the data corresponding to the terminal node is derived from the data corresponding to the starting node for the same data stream, in the case that only the second record information corresponding to the terminal node is collected, it can be considered that the first record information corresponding to the starting node exists in the target system, and further it can be considered that the corresponding data stream is normally completed.

[0083] In the third mode, in the case that it is determined based on the record information that the first record information is collected and the second record information is not collected, and the time length from the processing time of the target data corresponding to the first record information to the current time is greater than the preset time length, it is determined that the data state includes that the data stream corresponding to the target data is an abnormal data stream. The processing time of the target data corresponding to the first record information can be the time when the target data is input into the content effective system.

[0084] In this mode, in the case that only the first record information corresponding to the starting node is collected, since the second record information corresponding to the terminal node is missing, the completion of the corresponding data stream cannot be directly determined, and therefore the completion of the data stream is further determined based on the timeout of the collected first record information. In the case of timeout, it is considered that the data stream is not normally completed.

[0085] In the fourth mode, in the case that it is determined based on the record information that the first record information is collected and the second record information is not collected, and the time length from the processing time of the target data corresponding to the first record information to the current time is less than or equal to the preset time length, it is determined that the data state includes that the data stream corresponding to the target data is a normal data stream. The specific meaning of the processing time of the target data corresponding to the first record information can refer to the third mode, which will not be described here.

[0086] In this mode, in the case that only the first record information corresponding to the starting node is collected, since the second record information corresponding to the terminal node is missing, the completion of the corresponding data stream cannot be directly determined, and therefore the completion of the data stream is further determined based on the timeout of the collected first record information. In the case of timeout, it is considered that the data stream is not normally completed.

[0087] In the above-mentioned first to fourth modes, according to the specific content of the record information, it can be determined whether the first record information and the second record information are collected, and whether the time length from the processing time of the collected record information to the current time is timed out. In combination with the collection of the record information and the timeout, the data state of the target data can be quickly and accurately determined.

[0088] In an optional implementation, the data detection method provided by the present disclosure can further include: determining whether the consumption progress in the distributed storage structure is stored based on the record information.

[0089] Based on the record information, the subsequent consumption demand can be determined, and based on the consumption demand, the corresponding storage or non-storage operation of the consumption progress can be performed to meet the subsequent consumption demand. After storing the consumption progress, the record information and / or attribute information located before the progress in each shard of the distributed storage structure will not be consumed again, which can improve the overall consumption efficiency and fault tolerance.

[0090] Optionally, based on the record information, it is determined whether to store the consumption progress in the distributed storage structure, including: in the case of the above-mentioned mode one and mode three, the consumption progress of the corresponding shard in the distributed storage structure is stored; in the case of the above-mentioned mode two, in the case where the time length from the processing time of the target data corresponding to the second record information to the current time is greater than the preset time length, the consumption progress of the corresponding shard in the distributed storage structure is stored, and in the case where the time length from the processing time of the target data corresponding to the second record information to the current time is less than or equal to the preset time length, the consumption progress of the corresponding shard in the distributed storage structure is not stored; in the case of the above-mentioned mode four, the consumption progress of the corresponding shard in the distributed storage structure is not stored. The processing time of the target data corresponding to the second record information can be the time when the target data is output to the target system.

[0091] In the case of the above-mentioned mode one, both the first record information corresponding to the starting node and the second record information corresponding to the terminating node are collected, which is complete record information from the beginning to the end, and is a certain state, so that the consumed data (record information and / or attribute information) does not need to be consumed again, and thus the storage of the consumption progress makes the consumed data before the progress in the shard not be consumed repeatedly.

[0092] In the case of the above-mentioned mode two, the case where only the second record information corresponding to the terminating node is collected is usually caused by the out-of-order of data, and the subsequent consumption demand needs to be further determined according to the processing time of the target data corresponding to the second record information in the case of timeout. In the case of timeout, since for the same data stream, the data corresponding to the terminating node of the target system is derived from the data corresponding to the starting node, in the case of collecting the second record information, it can be determined that the corresponding first record information exists in the target system, and thus the timeout state in this mode is a certain state, so that the consumed data does not need to be consumed again, and thus the storage of the consumption progress makes the consumed data before the progress in the shard not be consumed repeatedly; in the case of not timeout, the missing first record information can be stored in the shard with delay, and if the consumption progress is stored before the first record information is stored in the shard, the data state of the target data can be incorrectly identified as an abnormal state, which can cause misjudgment, and thus the non-timeout state in this mode is an uncertain state, and thus the consumption progress is not stored to avoid misjudgment.

[0093] In the case of the third manner, in the case that only the first record information corresponding to the starting node is collected and the processing time of the target data corresponding to the first record information is overtime, that is, the second record information corresponding to the ending node is still missing until the overtime, in order to control the resources occupied by the record information, the missing first record information is no longer collected after the overtime, and the consumed data in the shard before the consumption progress is not required to be consumed again, so that the storage consumption progress is stored to avoid repeated consumption of the consumed data in the shard.

[0094] In the case of the fourth manner, in the case that only the first record information corresponding to the starting node is collected and the second record information corresponding to the ending node is missing, and the processing time of the target data corresponding to the first record information is not overtime, the missing second record information may be delayed in the shard, and if the consumption progress is stored before the second record information is stored in the shard, the data state of the target data may be incorrectly identified as an abnormal state, resulting in a false judgment. Therefore, in the case that the processing time is not overtime, the consumption progress is not stored to avoid the false judgment. In addition, in the case that the data detection device is restarted after being down, the data in the shard needs to be consumed again, and the consumption progress is not stored to avoid consumption abnormality after the restart.

[0095] Optionally, in the first to fourth manners, the judgment of whether the first record information and the second record information are collected, the judgment of whether the processing time of the target data corresponding to the collected record information is overtime, and the operation of determining the data state and whether to store the consumption progress in the distributed storage structure according to the judgment result can be performed multiple times. In an example, the operation can be performed periodically.

[0096] Optionally, in the case that the consumption progress in the distributed storage structure is stored, the data stored in the first storage structure and the second storage structure in the target operation unit can be cleared to release the resources of the first storage structure and the second storage structure.

[0097] Optionally, in the case that the consumption progress in the distributed storage structure is stored, the delay information of the corresponding data stream can also be stored and displayed, and the delay information of the data stream includes the difference between the processing time of the target data corresponding to the ending node and the processing time of the target data corresponding to the starting node in the data stream.

[0098] In an optional implementation, in the step S203, based on the data state, the effective state information of the target data in the target system is determined, including:

[0099] In the case that the data state includes that the data stream corresponding to the target data is an abnormal data stream, the effective state information is determined to include that the target data is in an abnormal effective state; in the case that the data state includes that the data stream corresponding to the target data is a normal data stream, the effective state information is determined to include that the target data is in a normal effective state.

[0100] Based on the data state, the effective state information accurately reflecting the effective state of the target data in the target system can be obtained.

[0101] In another optional embodiment, with reference to Figure 4 In step S203, the effective state information of the target data in the target system is determined based on the data state, including determining the bottleneck node based on the abnormal log, as follows:

[0102] In the case where the data state includes that the data flow corresponding to the target data is an abnormal data flow, the abnormal log of each processing node (or processing operator) in the target system is obtained (the storage form of the abnormal log is shown in Figure 6 According to the abnormal log, the bottleneck node in each processing node is determined, and the effective state information is determined based on the bottleneck node. The effective state information includes the identification information of the bottleneck node, and the bottleneck node is the processing node causing the data flow to be an abnormal data flow.

[0103] In the case where the data state includes that the data flow corresponding to the target data is an abnormal data flow, the bottleneck node causing the abnormality can be further determined according to the abnormal log, so as to take subsequent measures.

[0104] The processing node can be a processing module or a processing instance inside the processing module. In an example, the bottleneck module can be determined in each processing module according to the abnormal log, and then the bottleneck instance can be determined in each instance of the bottleneck module according to the abnormal log.

[0105] The abnormal log records the abnormal information of each processing node. According to the topological relationship between the processing nodes, the last abnormal processing node has a greater impact on the data flow, so the last processing node recorded in the abnormal log can be taken as the bottleneck node.

[0106] In an optional embodiment, the data detection method provided by the present disclosure further includes:

[0107] The data processing information of the target data in the target system is obtained, and the data aging index of each processing node in the target system is determined. According to the data processing information and the data aging index, the bottleneck node is determined in each processing node. Based on the bottleneck node, the effective state information is determined. The effective state information includes the identification information of the bottleneck node, and the bottleneck node is the processing node causing the data flow to be an abnormal data flow.

[0108] The data processing information can include the hardware identification information of the processing node and the processing time of processing the target data. Whenever the target data is processed by a new processing node, the hardware identification information of the processing node and the processing time of processing the target data can be added to the data processing information.

[0109] The specific meaning of the bottleneck node and specific examples of determining the bottleneck node can refer to the related content above, and will not be repeated here.

[0110] The data processing information can reflect the characteristics of the target system processing data, and the data aging index can reflect the characteristics of the delay of the data of the processing node. The way of determining the bottleneck node according to the data processing information and the data aging index is more targeted to the personalized processing characteristics of the target system, and can quickly locate the bottleneck node based on the delay characteristics of the data.

[0111] Optionally, the data aging index of each module in the target system is determined, including:

[0112] According to the data processing information, the topological relationship between each processing node is determined; according to the topological relationship, the upstream storage structure of each processing node is determined; for each processing node, the time information of the first target data and the last target data in the upstream storage structure of the processing node is obtained; according to the time information of the first target data and the last target data, the data aging index of the processing node is determined. The upstream storage structure is used to store the data sent by the previous processing node to the processing node.

[0113] In some embodiments, according to the data processing information, the topological relationship between each processing node is determined, including: through the topology analysis module, the hardware identification information in the data processing information is analyzed and clustered to obtain data topology information, and the topological relationship between each processing node is determined according to the data topology information. The data topology information can be in the form of a directed acyclic graph (DAG).

[0114] In the process of determining the topological relationship between each processing node according to the data topology information, the actually determined topological relationship may change on the basis of the original topological relationship set or user-defined, and in an example, the topological inversion may occur as shown in the figure. Figure 7 The direction of the arrow is reversed as a whole.

[0115] The topology analysis module can also determine and output tag information according to the hardware identification information, and determine the query rate per second and delay information between processing nodes according to the processing time in the data processing information.

[0116] The target data can be input to and output to the upstream storage structure according to the set time sequence. The first target data in the upstream storage structure can be the latest data input into that upstream storage structure, and the last target data can be the oldest data input into that upstream storage structure, which is also the data that will be output to that upstream storage structure. The time information of the target data can be the time when the target data is input to the target system (i.e., the select time, hereinafter referred to as system time).

[0117] Figure 7 The calculation principle of the data aging index in the processing module (specifically module b) is shown, with reference to... Figure 7 Determine the difference between the system time corresponding to the first target data and the system time corresponding to the last target data in the upstream storage structure of module b (e.g., Figure 7 The absolute value of the Δ system time shown is used as the data aging index for module b.

[0118] Optionally, the data aging metric can be calculated based on a streaming computing framework (such as Flink). The streaming computing framework can use the high / low watermark information of each processing node. This high / low watermark information corresponds to the system time of the first target data and the system time of the last target data, respectively, and can then be referenced as follows: Figure 8 The data aging index is calculated in the manner shown.

[0119] Based on the above methods, the embodiments of this disclosure can analyze and determine the topological relationship between each processing node in real time, and can achieve good support for customized topological relationships. Secondly, based on data aging indicators and aging thresholds, the data aging status of each node can be quickly and accurately determined, thereby improving the efficiency and accuracy of locating bottleneck nodes that affect the Activity on Vertex Network (AOV). At the same time, the data aging indicators determined based on time information can reflect the time accumulation of upstream storage structures. Compared with the traditional quantity accumulation, time accumulation can better reflect the timeliness of data flow. In addition, the existing methods for locating bottleneck nodes mainly detect the most downstream node causing congestion through backpressure detection. However, due to the differences in the size and traffic of storage structures between nodes, it is impossible to reflect the impact of bottleneck nodes on data timeliness. Moreover, at this time, serious data flow blockage has often already occurred. The method of locating bottleneck nodes based on data aging indicators can find the node that most affects the timeliness of data flow before backpressure occurs, thereby avoiding the above problems.

[0120] In another optional implementation, the data detection method provided in this disclosure further includes:

[0121] obtaining data processing information of target data in a target system; determining a topological relationship between each processing node in the target system according to the data processing information; determining whether the storage space of the upstream storage structure of each processing node has been fully occupied according to the topological relationship; for each processing node, in the case that the storage space of the upstream storage structure of the processing node has been fully occupied, determining that the processing node is a bottleneck node; in the case that the storage space of the upstream storage structure of the processing node has not been fully occupied, determining whether the processing node is a bottleneck node based on a data aging index of the processing node; determining effective state information based on the bottleneck node; wherein the effective state information includes identification information of the bottleneck node, and the bottleneck node is a processing node of an abnormal data flow.

[0122] The specific implementation of obtaining data processing information of target data in a target system and determining a topological relationship between each processing node in the target system can refer to the related content above, and will not be repeated here.

[0123] The specific implementation of determining whether the processing node is a bottleneck node based on a data aging index of the processing node can refer to the related content above, and will not be repeated here.

[0124] Based on the above method, the occupation of the storage space of the upstream storage structure of the processing node can intuitively show the accumulation of the upstream storage structure, and based on the accumulation, the bottleneck node can be quickly located.

[0125] Optionally, after the bottleneck node is determined, the bottleneck node can be expanded or repaired according to actual needs to overcome the adverse effects of the bottleneck node on the data flow.

[0126] In an optional embodiment, the data detection method provided by the present disclosure can further include: after the abnormal data flow is determined, the abnormal data flow is displayed.

[0127] In an example, the abnormal data flow can be sent to a client device of a business party, and the abnormal data flow is displayed through the client device of the business party, so as to alarm and intervene the abnormal data flow by the business party.

[0128] The inventors of the present disclosure analyzed the performance of the data detection method provided by the present disclosure and found that the data detection method provided by the present disclosure has relatively small resource and space complexity. In an example, assuming that the size of a single piece of data in a data stream is 1 kilobyte (including space amplification caused by a map structure), the preset time length of the data stream is 1 hour, the average traffic of the data stream is represented by a query rate per second, the query rate per second is 10,000 (W), and all map structures are stored in a memory, the data detection method provided by the present disclosure occupies a total of only about 30 megabytes (G). In an example, the write complexity of a single piece of data is O(1) to O(logN), and the complexity of the operation of performing processing timeout recording and storage consumption progress (compact) on a single piece of data can reach O(logN) after optimization of a message queue. The processing timeout recording and storage consumption progress can refer to the related content described above, for example, the judgment of whether the time length from the processing time of the target data corresponding to the collected record information to the current time is greater than the preset time length, and the operation of determining whether to store the consumption progress in the distributed storage structure based on the record.

[0129] The data detection method provided by the present disclosure can be applied to an overall vertical search content validation system (hereinafter referred to as a vertical search content validation system), and in the scenario of the vertical search content validation system, only about 100 megabytes (GB) of storage space and part of the message queue space are needed for buffering.

[0130] According to an embodiment of the present disclosure, the present disclosure further provides a data detection device, as shown in Figure 9 The device comprises a first information acquisition module 801, a data state determination module 802, and a first validation state determination module 803.

[0131] The first information acquisition module 801 is configured to acquire record information of target data in a target system. The record information comprises first record information corresponding to a starting node of the target system and / or second record information corresponding to a terminal node of the target system.

[0132] The data state determination module 802 is configured to determine the data state of the target data based on the record information.

[0133] The first validation state determination module 803 is configured to determine the validation state information of the target data in the target system based on the data state.

[0134] Optionally, the data state determination module 802 comprises an allocation sub-module, a first storage sub-module, and a data state determination sub-module.

[0135] The allocation sub-module is configured to allocate the target operation unit to the target data based on the identification information of the target data; the first storage sub-module is configured to store the record information in the first storage structure of the target operation unit based on the identification information; and the data state determination sub-module is configured to determine the data state of the target data based on the record information stored in the first storage structure by using the target operation unit.

[0136] Optionally, the data state determination module 802 further includes an information acquisition sub-module and a second storage sub-module.

[0137] The information acquisition sub-module is configured to acquire attribute information of the target data in the target system; and the second storage sub-module is configured to store the attribute information in the second storage structure of the target operation unit based on time information in the attribute information.

[0138] The data state determination sub-module is specifically configured to read the attribute information from the second storage structure based on a reading time sequence of the second storage structure; read corresponding record information from the first storage structure based on the read attribute information; and determine the data state of the target data based on the record information.

[0139] In an optional implementation, the data state determination module 802 is specifically configured to determine that the data state includes that the data stream corresponding to the target data is a normal data stream in a case where it is determined based on the record information that the first record information and the second record information of the target data are collected.

[0140] In another optional implementation, the data state determination module 802 is specifically configured to determine that the data state includes that the data stream corresponding to the target data is a normal data stream in a case where it is determined based on the record information that the second record information of the target data is collected and the first record information is not collected.

[0141] In yet another optional implementation, the data state determination module 802 is specifically configured to determine that the data state includes that the data stream corresponding to the target data is an abnormal data stream in a case where it is determined based on the record information that the first record information is collected, the second record information is not collected, and a time length from a processing time of the target data corresponding to the first record information to a current time is greater than a preset time length.

[0142] In still another optional implementation, the data state determination module is specifically configured to determine that the data state includes that the data stream corresponding to the target data is a normal data stream in a case where it is determined based on the record information that the first record information is collected, the second record information is not collected, and a time length from a processing time of the target data corresponding to the first record information to a current time is less than or equal to a preset time length.

[0143] In an optional implementation, the first effective state determining module 803 is specifically configured to: in a case where the data state includes that the data stream corresponding to the target data is an abnormal data stream, determine that the effective state information includes that the target data is in an abnormal effective state; and in a case where the data state includes that the data stream corresponding to the target data is a normal data stream, determine that the effective state information includes that the target data is in a normal effective state.

[0144] In another optional implementation, the first effective state determining module 803 includes a log obtaining sub-module, a bottleneck determining sub-module, and an effective state determining sub-module.

[0145] The log obtaining sub-module is configured to: in a case where the data state includes that the data stream corresponding to the target data is an abnormal data stream, obtain abnormal logs of each processing node in the target system; the bottleneck determining sub-module is configured to: according to the abnormal logs, determine a bottleneck node in each processing node; the bottleneck node is a processing node that causes the data stream to be an abnormal data stream; and the effective state determining sub-module is configured to: based on the bottleneck node, determine that the effective state information includes that the target data is in an abnormal effective state.

[0146] In an optional implementation, the data detection apparatus provided by the present disclosure further includes an index determining module, a first bottleneck determining module, and a second effective state determining module.

[0147] The index determining module is configured to: obtain data processing information of the target data in the target system, and determine data aging indexes of each processing node in the target system; the first bottleneck determining module is configured to: according to the data aging indexes, determine a bottleneck node in each processing node; the bottleneck node is a processing node that causes the data stream to be an abnormal data stream; and the second effective state determining module is configured to: based on the bottleneck node, determine that the effective state information includes that the target data is in an abnormal effective state.

[0148] Optionally, the index determining module is specifically configured to: determine a topological relationship between each processing node according to the data processing information; determine an upstream storage structure of each processing node according to the topological relationship; for each processing node, obtain time information of a first target data and a last target data in the upstream storage structure of the processing node; and determine a data aging index of the processing node according to the time information of the first target data and the last target data; wherein the upstream storage structure is configured to store data sent by a previous processing node to the processing node.

[0149] In another optional implementation, the data detection apparatus provided by the present disclosure further includes a second information obtaining module, a topology determining module, a space determining module, a second bottleneck determining module, and a third effective state determining module.

[0150] The second information acquisition module is configured to acquire data processing information of the target data in the target system; the topology determination module is configured to determine a topology relationship between each processing node in the target system according to the data processing information; the space determination module is configured to determine whether the storage space of the upstream storage structure of each processing node has been fully occupied according to the topology relationship; the second bottleneck determination module is configured to, for each processing node, determine that the processing node is a bottleneck node in a case where the storage space of the upstream storage structure of the processing node has been fully occupied; the bottleneck node is a processing node of the abnormal data flow; and the third effective state determination module is configured to determine, based on the bottleneck node, that the effective state information includes the target data in the abnormal effective state.

[0151] According to the disclosed embodiments, the present disclosure also provides a data detection system as Figure 9 The data detection system shown includes a framework layer (a core module of the data detection system), an aggregation layer, a presentation layer, and a storage layer. The present disclosure provides that the data detection device can be applied to the framework layer, and can specifically implement the steps of determining data states, determining bottleneck nodes, collecting data topology information, collecting abnormal logs, and the like, Figure 10 Only some steps are shown. In order to reduce storage costs, the data detection system can perform stream aggregation on record information, attribute information, data topology information, and the like in the aggregation layer through open source software (such as statsd, skywalking, and the like). The data detection system can present the record information, attribute information, and data topology information after stream aggregation in the presentation layer, and also present abnormal logs to implement log queries; the data detection system can store related data in the data detection system through storage units such as elastic search, redis (remote dictionary service), mongo, prometheus (system and service monitoring system), and the like in the storage layer, wherein redis can be used to cache key indicators in the data detection system, such as normal data and / or abnormal data in each data flow, mongo can be used to store meta information of the data flow, such as tag information used to mark the data flow and other associated information, and prometheus can be used to store the aggregated information at each moment as time series data, providing a data basis for querying and presenting according to the time latitude.

[0152] The functions of the modules, sub-modules, and units in the devices of the embodiments of the present disclosure can be referred to the corresponding descriptions in the above method embodiments, which will not be described here again.

[0153] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a non-transitory computer-readable storage medium, and a computer program product.

[0154] The electronic device provided by the present disclosure comprises at least one processor and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data detection method provided by any one of the embodiments of the present disclosure.

[0155] The non-transitory computer readable storage medium provided by the present disclosure stores computer instructions for enabling a computer to perform the data detection method provided by any one of the embodiments of the present disclosure.

[0156] The computer program product provided by the present disclosure comprises a computer program which, when executed by a processor, implements the data detection method provided by any one of the embodiments of the present disclosure.

[0157] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0158] As shown in ​ The device 1000 includes a computing unit 1001 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0159] Various components in the device 1000 are connected to the I / O interface 1005, including an input unit 1006, such as a keyboard, a mouse, etc., an output unit 1007, such as various types of displays, speakers, etc., a storage unit 1008, such as a magnetic disk, an optical disk, etc., and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0160] The computing unit 1001 can be various general purpose and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs various methods and processes described above. For example, in some embodiments, the above-described methods can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, portions or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded onto the RAM 1003 and executed by the computing unit 1001, one or more steps of the above-described methods can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the above-described methods by any other suitable means, such as by means of firmware.

[0161] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0162] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0163] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0165] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0166] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0167] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0168] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data detection method, comprising: obtaining record information of target data in a target system; the record information comprises first record information corresponding to a starting node of the target data in the target system and / or second record information corresponding to a terminal node of the target data in the target system; determining a data state of the target data based on the record information, comprising: in a case where, based on the record information, it is determined that the first record information is collected and the second record information is not collected, and a time length from a processing time of the target data corresponding to the first record information to a current time is greater than a preset time length, determining that the data state comprises an abnormal data flow of a data flow corresponding to the target data; determining effective state information of the target data in the target system based on the data state; wherein the determination of the effective state information of the target data in the target system based on the data state comprises: in a case where the data state comprises an abnormal data flow of a data flow corresponding to the target data, obtaining abnormal logs of each processing node in the target system; determining a bottleneck node in the each processing node according to the abnormal logs; the bottleneck node is a processing node causing the data flow to be an abnormal data flow; determining the effective state information based on the bottleneck node; wherein the effective state information comprises identification information of the bottleneck node.

2. The data detection method of claim 1, wherein, The determination of the data state of the target data based on the record information comprises: allocating a target operation unit for the target data based on identification information of the target data; storing the record information in a first storage structure of the target operation unit based on the identification information; determining the data state of the target data based on the record information stored in the first storage structure by using the target operation unit.

3. The data detection method of claim 2, wherein, The determination of the data state of the target data based on the record information further comprises: obtaining attribute information of the target data in the target system; storing the attribute information in a second storage structure in the target operation unit based on time information in the attribute information; The determination of the data state of the target data based on the record information stored in the first storage structure comprises: reading the attribute information from the second storage structure based on a reading sequence of the second storage structure; reading corresponding record information from the first storage structure based on the read attribute information; determining the data state of the target data based on the record information.

4. The data detection method according to any one of claims 1 to 3, wherein, The determination of the data state of the target data based on the record information comprises: in a case where, based on the record information, it is determined that the second record information is collected, determining that the data state comprises a normal data flow of a data flow corresponding to the target data.

5. The data detection method according to any one of claims 1 to 3, wherein, The determination of the data state of the target data based on the record information comprises: In a case where it is determined, based on the record information, that the first record information is collected and the second record information is not collected, and a time length from a processing time of the target data corresponding to the first record information to a current time is less than or equal to a preset time length, it is determined that the data state includes that the data stream corresponding to the target data is a normal data stream.

6. The data detection method according to any one of claims 1 to 3, wherein, The determining, based on the data state, of the effective state information of the target data in the target system includes: In a case where the data state includes that the data stream corresponding to the target data is an abnormal data stream, it is determined that the effective state information includes that the target data is in an abnormal effective state; In a case where the data state includes that the data stream corresponding to the target data is a normal data stream, it is determined that the effective state information includes that the target data is in a normal effective state.

7. The data detection method according to any one of claims 1 to 3, further comprising: obtaining data processing information of the target data in the target system, and determining data aging indexes of each processing node in the target system; determining a bottleneck node in the each processing node according to the data aging indexes; the bottleneck node is a processing node with an abnormal data stream; determining the effective state information based on the bottleneck node; the effective state information includes identification information of the bottleneck node.

8. The data detection method of claim 7, wherein, The determining of the data aging indexes of each module in the target system includes: determining a topology relationship between the each processing node according to the data processing information; determining an upstream storage structure of each processing node according to the topology relationship; for each processing node, obtaining time information of a first target data and a last target data in the upstream storage structure of the processing node; the upstream storage structure is used to store data sent by a previous processing node to the processing node; determining a data aging index of the processing node according to the time information of the first target data and the last target data.

9. The data detection method according to any one of claims 1 to 3, further comprising: obtaining data processing information of the target data in the target system; determining a topology relationship between the each processing node in the target system according to the data processing information; determining whether storage spaces of upstream storage structures of the each processing node have been fully occupied according to the topology relationship; for each processing node, in a case where the storage spaces of the upstream storage structure of the processing node have been fully occupied, determining that the processing node is a bottleneck node; the bottleneck node is a processing node with an abnormal data stream; determining the effective state information based on the bottleneck node; the effective state information includes identification information of the bottleneck node.

10. A data detection apparatus, comprising: a first information obtaining module configured to obtain record information of target data in a target system; the record information includes first record information corresponding to a starting node of the target data in the target system and / or second record information corresponding to a terminal node of the target data in the target system; a data state determination module, configured to determine a data state of the target data based on the record information; the data state determination module is specifically configured to: in a case where it is determined based on the record information that the first record information is collected and the second record information is not collected, and a time length from a processing time of the target data corresponding to the first record information to a current time is greater than a preset time length, determine that the data state includes that a data flow corresponding to the target data is an abnormal data flow; a first effective state determination module, configured to determine effective state information of the target data in the target system based on the data state; wherein the first effective state determination module includes: a log acquisition submodule, configured to acquire abnormal logs of each processing node in the target system in a case where the data state includes that the data flow corresponding to the target data is an abnormal data flow; a bottleneck determination submodule, configured to determine a bottleneck node in the each processing node according to the abnormal logs; the bottleneck node is a processing node causing the data flow to be an abnormal data flow; an effective state determination submodule, configured to determine the effective state information based on the bottleneck node; the effective state information includes identification information of the bottleneck node.

11. The data detection device of claim 10, wherein, the data state determination module includes: an allocation submodule, configured to allocate a target operation unit for the target data based on identification information of the target data; a first storage submodule, configured to store the record information in a first storage structure of the target operation unit based on the identification information; a data state determination submodule, configured to determine the data state of the target data based on the record information stored in the first storage structure by using the target operation unit.

12. The data detection device of claim 11, wherein, the data state determination module further includes: an information acquisition submodule, configured to acquire attribute information of the target data in the target system; a second storage submodule, configured to store the attribute information in a second storage structure in the target operation unit based on time information in the attribute information; the data state determination submodule is specifically configured to: read the attribute information from the second storage structure based on a reading time sequence of the second storage structure; read corresponding record information from the first storage structure based on the read attribute information; and determine the data state of the target data based on the record information.

13. The data detection device according to any one of claims 10 to 12, wherein, the data state determination module is specifically configured to: in a case where it is determined based on the record information that the second record information is collected, determine that the data state includes that the data flow corresponding to the target data is a normal data flow.

14. The data detection device according to any one of claims 10 to 12, wherein, the data state determination module is specifically configured to: in a case where it is determined based on the record information that the first record information is collected and the second record information is not collected, and a time length from a processing time of the target data corresponding to the first record information to a current time is less than or equal to a preset time length, determine that the data state includes that the data flow corresponding to the target data is a normal data flow.

15. The data detection device according to any one of claims 10 to 12, wherein, the first effective state determination module is specifically configured to: In a case where the data state comprises that the data flow corresponding to the target data is an abnormal data flow, the effective state information is determined to comprise that the target data is in an abnormal effective state; In a case where the data state comprises that the data flow corresponding to the target data is a normal data flow, the effective state information is determined to comprise that the target data is in a normal effective state.

16. The data detection apparatus according to any one of claims 10-12, further comprising: an index determination module configured to acquire data processing information of the target data in the target system, and determine data aging indexes of each processing node in the target system; a first bottleneck determination module configured to determine a bottleneck node in the each processing node according to the data aging indexes; the bottleneck node is a processing node with an abnormal data flow; a second effective state determination module configured to determine the effective state information based on the bottleneck node; the effective state information comprises identification information of the bottleneck node.

17. The data detection apparatus of claim 16, wherein, The index determination module is specifically configured to: acquire data processing information of the target data in the target system; determine a topology relationship between the each processing node according to the data processing information; determine an upstream storage structure of each processing node according to the topology relationship; for each processing node, acquire time information of a first target data and a last target data in the upstream storage structure of the processing node; and determine a data aging index of the processing node according to the time information of the first target data and the last target data; the upstream storage structure is configured to store data sent by a previous processing node to the processing node.

18. The data detection apparatus according to any one of claims 10-12, further comprising: a second information acquisition module configured to acquire data processing information of the target data in the target system; a topology determination module configured to determine a topology relationship between the each processing node in the target system according to the data processing information; a space determination module configured to determine whether storage spaces of the upstream storage structure of the each processing node have been fully occupied according to the topology relationship; a second bottleneck determination module configured to determine, for each processing node, that the processing node is a bottleneck node in a case where the storage spaces of the upstream storage structure of the processing node have been fully occupied; the bottleneck node is a processing node with an abnormal data flow; a third effective state determination module configured to determine the effective state information based on the bottleneck node; the effective state information comprises identification information of the bottleneck node.

19. An electronic device, comprising: at least one processor; and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data detection method in any one of claims 1-9. The computer instructions are used to enable the computer to perform the data detection method in any one of claims 1-9.

20. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, ​ 21. A computer program product comprising a computer program which, when executed by a processor, implements the data detection method according to any one of claims 1-9.

Images