Data accumulation detection method and device, and electronic equipment

Abstract

The application provides a data accumulation detection method, device and electronic equipment. The method comprises the following steps: determining target detection configuration information corresponding to a target type of a data transmission source to be detected based on the target type and pre-generated detection configuration information corresponding to multiple types; wherein the multiple types at least comprise a file system, a message queue and a database; obtaining a target detection value of the data transmission source to be detected based on the target detection configuration information; and determining whether the data transmission source to be detected has data accumulation based on the target detection value and pre-generated detection threshold values corresponding to the multiple types. In this way, when detecting data accumulation of the data transmission source, the detection configuration information and the detection threshold values corresponding to multiple data transmission source types are defined, the data accumulation of the multiple data transmission sources can be detected, and thus the operation of various boundary devices can be more comprehensively supervised by the supervision platform.

Description

Technical Field

[0001] This invention relates to the field of cross-network transmission and switching system management technology, and in particular to a data accumulation detection method, device, and electronic device. Background Technology

[0002] During cross-network data transmission, devices at the boundary exchange data through a secure channel. Data exchange may cause data accumulation at the source end due to device or network anomalies. Ideally, data transmission at the source end and data reception at the target end are synchronous and equivalent. If the target end malfunctions and cannot receive data normally, or if the boundary device in the intermediate channel malfunctions, the data sent at the source end will accumulate. If the accumulation reaches a certain level, it may cause system link interruption and communication accident.

[0003] Currently, traditional data accumulation detection methods detect only a single type of data source, limiting the monitoring platform's ability to oversee the operation of boundary devices. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data accumulation detection method, apparatus and electronic device to detect data accumulation at multiple data transmission sources, thereby enabling the monitoring platform to more comprehensively monitor the operation of various boundary devices.

[0005] In a first aspect, embodiments of the present invention provide a data accumulation detection method, the method comprising: determining the data transmission source to be detected based on the link connection status corresponding to each data transmission source in advance, and establishing a communication connection with the data transmission source to be detected;

[0006] Based on the target type of the data transmission source to be detected, and the detection configuration information corresponding to various pre-generated types, the target detection configuration information corresponding to the target type is determined; among them, the various types include at least: file system, message queue, and database;

[0007] Based on the target detection configuration information, obtain the target detection value at the source end of the data transmission to be detected;

[0008] Based on the target detection value and the detection thresholds corresponding to various pre-generated types, it is determined whether data accumulation occurs at the data transmission source of the data to be detected.

[0009] Furthermore, the detection configuration information corresponding to each of the various types is generated in the following way:

[0010] Based on the pre-acquired quantitative indicators configured for various types, detection configuration information corresponding to each type is generated.

[0011] Furthermore, the step of generating detection configuration information for each type based on pre-acquired quantitative indicators configured for various types includes:

[0012] Obtain the first quantitative metric of the user's file system configuration; wherein, the first quantitative metric includes: the number of remaining files and the total size of remaining files;

[0013] Based on each first quantitative indicator, generate the first detection configuration information corresponding to the file system;

[0014] Obtain the second quantitative metric configured by the user for the message queue; wherein the second quantitative metric includes: topic name and remaining consumption offset;

[0015] Based on each second quantitative indicator, generate the second detection configuration information corresponding to the message queue;

[0016] Obtain the third quantitative metric for the user's database configuration; the third quantitative metric includes: the number of rows of records to be processed and the total size of records to be processed.

[0017] Based on each third quantitative indicator, generate the corresponding third detection configuration information for the database.

[0018] Furthermore, based on the target detection configuration information, the steps for obtaining the target detection value at the source end of the data transmission to be detected include:

[0019] Based on the target detection configuration information and the preset time period, the target detection value of the data transmission source corresponding to each time period is obtained.

[0020] Furthermore, the link connection status includes: connected status and disconnected status; based on the pre-acquired link connection status corresponding to each data transmission source, the steps of determining the data transmission source to be detected and establishing a communication connection with the data transmission source to be detected include:

[0021] Based on the link connection status corresponding to each data transmission source end obtained in advance, determine the target data transmission source end corresponding to the connected status;

[0022] The target data transmission source is identified as the data transmission source to be detected, and a communication connection is established with the data transmission source to be detected.

[0023] Furthermore, the detection thresholds for each type include: the number of times thresholds corresponding to the time period, and the indicator thresholds corresponding to the quantification indicators; wherein, the indicator thresholds include at least one of the following: indicator quantity thresholds and indicator size thresholds.

[0024] Based on the target detection value and the pre-generated detection thresholds for various types, the steps to determine whether data accumulation occurs at the data transmission source include:

[0025] Based on the target type of the data transmission source corresponding to the target detection value, determine the target detection threshold corresponding to the target type; wherein, the target detection threshold includes: target count threshold and target index threshold;

[0026] The target detection value is compared with the target index threshold to obtain the comparison result;

[0027] If the comparison result indicates that the target detection value is greater than the target indicator threshold, obtain the first target detection value corresponding to the next time period. If the first target detection value is greater than the target detection value, record the number of times corresponding to the time period as 1.

[0028] The cumulative count of the number of times corresponding to the time period is calculated. If the number of times corresponding to the time period is not greater than the target count threshold, the first target detection value is used as the new target detection value. The step of comparing the target detection value with the target index threshold and obtaining the comparison result is repeated until the number of times corresponding to the time period is greater than the target count threshold.

[0029] Generate alarm information corresponding to the data transmission source of the data to be detected; the alarm information is used to indicate that data accumulation has occurred at the data transmission source of the data to be detected.

[0030] Furthermore, after generating the alarm information corresponding to the data transmission source to be detected, the following steps are performed:

[0031] Based on the pre-obtained link connection information corresponding to the source end of the data transmission to be detected, the target boundary device and the target data transmission destination end corresponding to the source end of the data transmission to be detected are identified as the target abnormal device to be detected.

[0032] Secondly, embodiments of the present invention provide a data accumulation detection device, the device comprising:

[0033] The first determining module is used to determine the data transmission source to be detected based on the link connection status corresponding to each data transmission source in advance, and to establish a communication connection with the data transmission source to be detected.

[0034] The second determining module is used to determine the target detection configuration information corresponding to the target type based on the target type of the data transmission source and the detection configuration information corresponding to the various types generated in advance; wherein, the various types include at least: file system, message queue, and database;

[0035] The acquisition module is used to acquire the target detection value of the data source from the target detection configuration information.

[0036] The judgment module is used to determine whether there is data accumulation at the data transmission source of the data to be detected, based on the target detection value and the detection thresholds corresponding to various pre-generated types.

[0037] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of any of the above methods.

[0038] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the steps of any of the methods described above.

[0039] This invention provides a data accumulation detection method, apparatus, and electronic device. The method includes: determining target detection configuration information corresponding to the target type based on the target type of the data transmission source to be detected and pre-generated detection configuration information corresponding to various types; wherein, the various types include at least: file system, message queue, and database; obtaining the target detection value of the data transmission source to be detected based on the target detection configuration information; and determining whether data accumulation occurs at the data transmission source to be detected based on the target detection value and pre-generated detection thresholds corresponding to various types. In this method, when detecting data accumulation at the data transmission source, by customizing the detection configuration information and detection thresholds corresponding to various data transmission source types, data accumulation at various data transmission source sources can be detected, thereby enabling the monitoring platform to more comprehensively monitor the operation of various boundary devices.

[0040] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0042] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1A flowchart of a data piling detection method provided in an embodiment of the present invention;

[0044] Figure 2 A flowchart of another data accumulation detection method provided in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram illustrating a process for detecting data accumulation at multiple data transmission sources, provided by an embodiment of the present invention.

[0046] Figure 4 This is a schematic diagram of the structure of a data accumulation detection device provided in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the structure of a data accumulation detection system provided in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] For cross-network data transmission, data sending devices (i.e., data transmission source) and data receiving devices (i.e., data transmission destination) exchange data through boundary devices (abstractly understood as secure channels). Data exchange may experience data backlog due to device or network anomalies. Ideally, data transmission at the source and data reception at the destination are synchronous and equivalent. However, if the destination fails to receive data normally, or if the boundary device in the intermediate channel malfunctions, the data sent from the source will accumulate. If this accumulation reaches a certain level, it may cause a link interruption in the switching system, resulting in a communication accident. Currently available technologies detect only a single data source type and judge based solely on data quantity, failing to support data backlog detection for multiple transmission sources (equivalent to multiple transmission methods), thus limiting device monitoring.

[0051] Based on this, embodiments of the present invention provide a data accumulation detection method, apparatus, and electronic device, which can be applied to applications that require monitoring of the operation of boundary devices.

[0052] To facilitate understanding of this embodiment, a detailed description of the method for monitoring and detecting the operational status of boundary equipment disclosed in this embodiment of the invention will be provided first, such as... Figure 1As shown, the method includes the following steps:

[0053] Step S102: Based on the link connection status corresponding to each data transmission source obtained in advance, determine the data transmission source to be detected and establish a communication connection with the data transmission source to be detected.

[0054] The technical scenario applied in this application is mainly the operation of switching equipment used in the process of supervising cross-network data transmission (equivalent to cross-network switching tasks). The switching equipment typically includes data sending devices, border devices, and data receiving devices. In actual implementation, multiple data transmission sources (equivalent to the aforementioned data sending devices), multiple border devices, and multiple data transmission destinations can jointly form a cross-network data switching device connected to a cross-network data switching system (usually a server that can manage the connection relationships between various devices). The cross-network data switching system can store the link connection information and link connection status corresponding to each data sending device. In fact, the cross-network data switching system can configure corresponding border devices and corresponding data receiving devices for each data sending device. For example, for a certain data sending device, based on its sending request, it can match the border devices and data receiving devices that can respond to and process the sending request. Based on the matching result, the link connection information and link connection status corresponding to the data receiving device are generated and maintained. Therefore, each link connection information can include a data sending device, as well as the target border device and target data receiving device corresponding to the data sending device. Each link connection status can be either connected or disconnected, depending on the connection status of the corresponding data sending device.

[0055] In practice, it can receive the link connection status of each data transmission source sent from the cross-network data exchange system. Based on whether the data transmission source is connected normally, that is, based on whether the link connection status is connected, the data transmission source to be detected can be selected (in fact, data accumulation detection is not necessary for unconnected data transmission sources). The link connection status of the data transmission source to be detected should be connected. After the data transmission source to be detected is determined, a communication connection can be established with the data transmission source to be detected.

[0056] Step S104: Based on the target type of the data transmission source to be detected and the detection configuration information corresponding to the various types generated in advance, determine the target detection configuration information corresponding to the target type; wherein, the various types include at least: file system, message queue, and database.

[0057] Depending on the data transmission method, the above data transmission source can be of various types, such as file system, message queue, database, etc.

[0058] The detection configuration information corresponding to the above-mentioned various types can be generated in the following ways:

[0059] Based on the pre-acquired quantitative indicators configured for various types, detection configuration information corresponding to each type is generated.

[0060] In actual implementation, after obtaining the data transmission source to be detected, it can be matched with these pre-set types to obtain the matching target type. Then, the target detection configuration information corresponding to the target type is filtered out from the detection configuration information corresponding to each type.

[0061] Step S106: Based on the target detection configuration information, obtain the target detection value at the source end of the data transmission to be detected.

[0062] The target detection value obtained above from the source end of the data transmission to be detected can actually be understood as the detection value corresponding to the target detection configuration information. In actual implementation, assuming that the target type of the source end of the data transmission to be detected is a file system, the source end of the data transmission to be detected is file system A, and the target detection configuration information above is the remaining file size, then the target detection value of file system A (that is, the remaining file size value) obtained is equivalent to how much file size is left in file system A. For example, if the remaining file size in file system A is 5G, then the corresponding target detection value above is 5G.

[0063] Step S108: Based on the target detection value and the detection thresholds corresponding to various types generated in advance, determine whether there is data accumulation at the data transmission source end to be detected.

[0064] The detection thresholds for the various types mentioned above can be generated in the following ways:

[0065] Obtain the quantization threshold indicators configured by the user for each preset type, as well as the information provided for each quantization threshold indicator;

[0066] Based on each quantization threshold indicator and each piece of provided information, various types of detection thresholds are generated.

[0067] In actual implementation, the detection threshold can be customized; the data accumulation detection threshold for different types of transmission methods can be customized, and the threshold conditions for generating alarms can be configured according to business information.

[0068] Once the target detection value is obtained, it can be matched with these pre-set multiple types to obtain the matching target type. Then, from the detection thresholds corresponding to each type, the target detection threshold corresponding to that target type is selected.

[0069] By comparing the target detection value with the target detection threshold, it can be determined whether there is data accumulation at the source of the data to be detected. For example, if the target detection value is greater than the target detection threshold, it can be determined that there is data accumulation at the source of the data to be detected.

[0070] This invention provides a data accumulation detection method, apparatus, and electronic device. The method includes: determining target detection configuration information corresponding to the target type based on the target type of the data transmission source to be detected and pre-generated detection configuration information corresponding to various types; wherein the various types include at least: file system, message queue, and database; obtaining the target detection value of the data transmission source to be detected based on the target detection configuration information; and determining whether data accumulation occurs at the data transmission source to be detected based on the target detection value and pre-generated detection thresholds corresponding to various types. In this method, when detecting data accumulation at the data transmission source, by customizing the detection configuration information and detection thresholds corresponding to various data transmission source types, data accumulation at various data transmission source sources can be detected, thereby enabling the monitoring platform to more comprehensively monitor the operation of various boundary devices.

[0071] This invention also provides another data accumulation detection method, which is implemented based on the method in the above embodiments; this method focuses on describing the process of determining whether data accumulation occurs at the data transmission source end to be detected, such as... Figure 2 As shown, the method includes the following steps:

[0072] Step S202: Based on the link connection status corresponding to each data transmission source end obtained in advance, determine the target data transmission source end corresponding to the connected status;

[0073] Step S204: The target data transmission source is identified as the data transmission source to be detected, and a communication connection is established with the data transmission source to be detected.

[0074] Step S206: Based on the target type of the data transmission source and the detection configuration information corresponding to the various types generated in advance, determine the target detection configuration information corresponding to the target type.

[0075] In practice, the types of data transmission sources mentioned above can include file systems, message queues, and databases;

[0076] The detection configuration information corresponding to the above-mentioned various types can be implemented through the following steps one through six:

[0077] Step 1: Obtain the first quantitative indicator of the user's file system configuration; the first quantitative indicator includes: the number of remaining files and the total size of remaining files;

[0078] Step 2: Generate the first detection configuration information for the file system based on each first quantitative indicator;

[0079] In actual implementation, for file system types, the aforementioned first detection configuration information can be understood as the custom-configured detection information during file exchange tasks; specifically, the content of the aforementioned first detection configuration information is the same as that of the first quantitative indicator.

[0080] Step 3: Obtain the second quantitative metric configured by the user for the message queue; the second quantitative metric includes: topic name and remaining consumption offset;

[0081] Step 4: Generate the second detection configuration information corresponding to each second quantification indicator for the message queue;

[0082] In practical implementation, the aforementioned message queue can be understood as a container for storing messages during message transmission, typically in a distributed publish-subscribe messaging system. The topic name (Topic) is actually a subscription topic (message topic, which can be understood as a message category) in the distributed publish-subscribe messaging system, and the messages are stored in partitions of the Topic. The remaining consumption offset can be understood as the number of unconsumed messages recorded when a consumer consumes messages in a partition. For this type of message queue, the second detection configuration information can be understood as custom-configured detection information during message queue exchange tasks; specifically, the content of the second detection configuration information is the same as that of the second quantitative indicator.

[0083] Step 5: Obtain the third quantitative indicator configured by the user for the database; the third quantitative indicator includes: the number of rows of records to be processed and the total size of records to be processed.

[0084] Step 6: Generate the third detection configuration information corresponding to the message queue based on each third quantification indicator.

[0085] In practice, data is stored in the database as rows of tables. The unit of database exchange is the table, and data in the table is processed row by row. Therefore, the number of remaining unprocessed rows mentioned above can be understood as the number of remaining unprocessed rows in the table (the number of rows in the table that have not been processed); the total size of the remaining unprocessed records mentioned above can be understood as the file system space occupied by the database table. For this type of database, the third detection configuration information mentioned above can be understood as the detection information customized during database exchange tasks; specifically, the content of the third detection configuration information is the same as that of the third quantitative indicator.

[0086] Step S208: Based on the target detection configuration information and the preset time period, obtain the target detection value of the data transmission source corresponding to each time period.

[0087] In actual implementation, the target type of the data transmission source to be detected may be a file system, a message queue, or a database. Therefore, the corresponding target detection configuration information mentioned above may also be one of the first detection configuration information, the second detection configuration information, or the third detection configuration information.

[0088] The preset time period mentioned above can be understood as a time interval (i.e., acquisition frequency). For example, if the time interval is 1 minute, then a corresponding target detection value can be acquired every 1 minute.

[0089] Specifically, distributed processing can be used to periodically obtain the remaining quantity and size of data at the cross-network data transmission source end, so as to periodically detect and monitor the cross-network data transmission device, realize real-time monitoring of whether there is data accumulation at the cross-network data transmission source end, and thus ensure the timeliness of data accumulation detection.

[0090] Step S210: Determine the target detection threshold corresponding to the target type based on the target type of the data transmission source corresponding to the target detection value; wherein, the target detection threshold includes: target count threshold and target index threshold;

[0091] In actual implementation, the detection thresholds for each type include: the number of times thresholds corresponding to the time period, and the index thresholds corresponding to the quantification index; wherein, the index thresholds include at least one of the following: index quantity thresholds and index size thresholds;

[0092] Specifically, the detection thresholds corresponding to the above-mentioned various types include: a first detection threshold for the file system, a second detection threshold for the message queue, and a third detection threshold for the database; wherein, the first detection threshold includes: a first number threshold corresponding to the time period, and a first indicator threshold corresponding to the first quantification indicator; wherein, the first indicator threshold includes: a first quantity threshold and a first size threshold; the second detection threshold includes: a second number threshold corresponding to the time period, and a second indicator threshold corresponding to the second quantification indicator; wherein, the second indicator threshold includes: a second quantity threshold; the third detection threshold includes: a third number threshold corresponding to the time period, and a third indicator threshold corresponding to the third quantification indicator; wherein, the third indicator threshold includes: a third quantity threshold and a second size threshold;

[0093] In actual implementation, assuming the target type is a file system, the target count threshold corresponding to the target type is the first count threshold, and the target indicator threshold is the first quantity threshold and the first size threshold.

[0094] Step S212: Compare the target detection value with the target index threshold to obtain the comparison result;

[0095] Step S214: If the comparison result indicates that the target detection value is greater than the target index threshold, obtain the first target detection value corresponding to the next time period; if the first target detection value is greater than the target detection value, record the number of times corresponding to the time period as 1.

[0096] In actual implementation, taking the file system as the target type as an example, the target detection values ​​mentioned above can include: the number of remaining files and the total size of remaining files. The number of remaining files is compared with the first quantity threshold, and the total size of remaining files is compared with the first size threshold. Two comparison results can be obtained. If both of these results meet the preset conditions, such as the total size of remaining files being greater than the first size threshold and the number of remaining files being greater than the first quantity threshold, the first target detection value corresponding to the next time period can be obtained. At this time, the first target detection value may be greater than the latest obtained target detection value (that is, the latest obtained target detection value before obtaining the first target detection value of the next time period) or less than the latest obtained target detection value. That is, the first target detection value may be continuously increasing or decreasing.

[0097] In practical implementation, in order to improve the accuracy of data accumulation judgment and thus the accuracy of equipment monitoring, it is possible not only to determine whether the remaining quantity and size exceed the preset indicator threshold, but also to determine whether the cumulative number of consecutive increases (i.e., the cumulative number of times from the first time the preset indicator threshold is exceeded to the subsequent time periods) exceeds the preset number (i.e., the number threshold). In fact, since the time period is a preset time interval, it is also equivalent to judging whether the duration of consecutive increases exceeds the preset duration. This can avoid the poor accuracy caused by judging based solely on the data quantity in the existing technology.

[0098] Specifically, starting from the first comparison result that meets the second preset condition (i.e., indicating that the target detection value is greater than the target index threshold), the number of time cycles can be recorded as 1. If comparison results that meet the second preset condition are obtained continuously (i.e., indicating that the current target detection value is greater than the previous target detection value), the record can be accumulated on the previous cycle count data. For example, after the first comparison result that meets the second preset condition is obtained, if the comparison result obtained in the next time cycle meets the second preset condition, the number of time cycles can be recorded as 2.

[0099] Step S216: Accumulate the number of times corresponding to the time period. If the number of times corresponding to the time period is not greater than the target number threshold, take the first target detection value as the new target detection value, and repeat the step of comparing the target detection value with the target index threshold to obtain the comparison result until the number of times corresponding to the time period is greater than the target number threshold.

[0100] In actual implementation, if the number of times corresponding to the time period, that is, the cumulative value of the number of times recorded in the above time period, is not greater than the preset target number threshold, which is equivalent to the cumulative number of continuously increasing times not exceeding the preset number (which is the same as the number threshold corresponding to the time period), and also equivalent to the duration of continuously increasing times not exceeding the preset duration, then the above steps S212-S216 can be repeated until the number of times corresponding to the time period is greater than the target number threshold.

[0101] Step S218: Generate alarm information corresponding to the data transmission source of the data to be detected; wherein, the alarm information is used to indicate that data accumulation has occurred at the data transmission source of the data to be detected.

[0102] In practice, if both the target detection value and the number of occurrences within the time period exceed the target number threshold, an alarm message will be generated at the data transmission source to alert regulatory personnel that data accumulation has occurred at the data transmission source. This allows for timely maintenance and inspection.

[0103] Specifically, it can promptly detect data accumulation problems and generate accurate data accumulation alarm information, thereby enabling timely detection and handling of equipment anomalies.

[0104] Step S220: Based on the link connection information corresponding to the source end of the data transmission to be detected obtained in advance, the target boundary device and the target data transmission destination end corresponding to the source end of the data transmission to be detected are determined as the target abnormal device to be detected.

[0105] In practice, the link connection information of the data transmission source to be detected that is in a connected state can be obtained in advance. This link connection information can include the target boundary device and the target data transmission destination corresponding to the data transmission source to be detected. Therefore, after generating accurate data accumulation alarm information, the target abnormal device to be detected can be identified by obtaining the link connection information of the data transmission source to be detected that has already accumulated data. This greatly ensures the rapid location and recovery of abnormal boundary devices and abnormal data receiving devices (data transmission destination).

[0106] The data accumulation detection method provided in the above embodiments adopts specific data accumulation detection configuration information and detection thresholds for different transmission methods. In addition to determining the amount of data, it can also determine the size and duration of data accumulation to generate accurate data accumulation information alarm information, ensuring timely detection and recovery of equipment failures. At the same time, it can realize data accumulation detection of multiple data source types such as files, databases, and message queues, improving the comprehensiveness and accuracy of supervision.

[0107] To better understand the above embodiments, please refer to, for example... Figure 3The diagram illustrates a process for detecting data accumulation at multiple data transmission sources. Figure 3 The system enables distributed data processing. The source data acquisition service and the data analysis service are equivalent to two microservices, with data transmitted between them via API (Application Programming Interface). The source data acquisition service can periodically collect the data accumulation status from the source and transmit it to the data analysis service for real-time analysis via API. Figure 3 The incrementing period in the time period can be understood as the threshold number of times corresponding to the time period.

[0108] This embodiment provides a data accumulation detection method. Based on preset data source connection information (equivalent to link connection information and link connection status) and frequency (time period), it periodically acquires the remaining quantity and size of data at the cross-network data transmission source. Different information is acquired depending on the cross-network data transmission method. For file transmission, the indicators are the number of remaining files and the total size of remaining files; for message queue transmission, the indicators are the topic name and the remaining unconsumed offset; and for database transmission, the indicators are the number of remaining pending records and the total size of remaining pending records. The method determines whether the collected indicator data meets the preset data accumulation determination conditions (thresholds) by using block judgment. That is, when the relevant indicators of the data at the data transmission source reach the threshold and continuously increase within the threshold period, and the cumulative number of continuous increases exceeds the preset number threshold, and the duration of continuous increases exceeds the preset duration threshold, then it is determined that data accumulation has occurred at the data transmission source.

[0109] This invention provides a schematic diagram of the structure of a data accumulation detection device, as shown in the embodiment of the invention. Figure 4 As shown, the device includes: a first determining module 40, used to determine the data transmission source to be detected based on the link connection status corresponding to each data transmission source in advance, and to establish a communication connection with the data transmission source to be detected; a second determining module 41, used to determine the target detection configuration information corresponding to the target type based on the target type of the data transmission source to be detected and the detection configuration information corresponding to the various types in advance; wherein, the various types include at least: file system, message queue, and database; an acquiring module 42, used to acquire the target detection value of the data transmission source to be detected based on the target detection configuration information; and a judging module 43, used to judge whether data accumulation occurs at the data transmission source to be detected based on the target detection value and the detection thresholds corresponding to the various types in advance.

[0110] This invention provides a data accumulation detection device. The device includes: determining target detection configuration information corresponding to a target type based on the target type of the data transmission source to be detected and pre-generated detection configuration information corresponding to various types; wherein the various types include at least: file system, message queue, and database; obtaining a target detection value of the data transmission source to be detected based on the target detection configuration information; and determining whether data accumulation occurs at the data transmission source to be detected based on the target detection value and pre-generated detection thresholds corresponding to various types. In this device, when detecting data accumulation at the data transmission source, by customizing the detection configuration information and detection thresholds corresponding to various data transmission source types, data accumulation at multiple data transmission source sources can be detected, thereby enabling the monitoring platform to more comprehensively monitor the operation of various boundary devices.

[0111] Furthermore, the data accumulation detection device also includes a generation module, which generates detection configuration information for each type based on pre-acquired quantitative indicators configured for various types.

[0112] Furthermore, the generation module is also used to: obtain the first quantitative indicator of the user's configuration of the file system; wherein the first quantitative indicator includes: the number of remaining files and the total size of remaining files; and generate the first detection configuration information corresponding to the file system based on each first quantitative indicator; obtain the second quantitative indicator of the user's configuration of the message queue; wherein the second quantitative indicator includes: the topic name and the remaining consumption offset; and generate the second detection configuration information corresponding to the message queue based on each second quantitative indicator.

[0113] Obtain the third quantitative metric for the user's database configuration; the third quantitative metric includes: the number of rows of records to be processed and the total size of records to be processed.

[0114] Based on each third quantitative indicator, generate the third detection configuration information corresponding to the message queue.

[0115] Furthermore, the acquisition module is also used to: acquire the target detection value of the data transmission source corresponding to each time period based on the target detection configuration information and the preset time period.

[0116] Furthermore, the link connection status includes: connected state and disconnected state; the first determining module is also used for:

[0117] Based on the link connection status corresponding to each data transmission source end obtained in advance, determine the target data transmission source end corresponding to the connected status;

[0118] The target data transmission source is identified as the data transmission source to be detected, and a communication connection is established with the data transmission source to be detected.

[0119] Furthermore, the detection thresholds for each type include: the number of times thresholds corresponding to the time period, and the indicator thresholds corresponding to the quantification indicators; wherein, the indicator thresholds include at least one of the following: indicator quantity thresholds and indicator size thresholds.

[0120] The judgment module is also used to: determine the target detection threshold corresponding to the target type based on the target type of the data transmission source corresponding to the target detection value; wherein, the target detection threshold includes: target count threshold and target index threshold;

[0121] The target detection value is compared with the target index threshold to obtain the comparison result;

[0122] If the comparison result indicates that the target detection value is greater than the target index threshold, obtain the first target detection value corresponding to the next time period. If the first target detection value is greater than the target detection value, record the number of times corresponding to the time period as 1.

[0123] The cumulative count of the number of times corresponding to the time period is calculated. If the number of times corresponding to the time period is not greater than the target count threshold, the first target detection value is used as the new target detection value. The step of comparing the target detection value with the target index threshold and obtaining the comparison result is repeated until the number of times corresponding to the time period is greater than the target count threshold.

[0124] Generate alarm information corresponding to the data transmission source of the data to be detected; the alarm information is used to indicate that data accumulation has occurred at the data transmission source of the data to be detected.

[0125] Furthermore, the third determining module is used for:

[0126] Based on the pre-obtained link connection information corresponding to the source end of the data transmission to be detected, the target boundary device and the target data transmission destination end corresponding to the source end of the data transmission to be detected are identified as the target abnormal device to be detected.

[0127] The data piling detection device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned data piling detection method embodiment. For the data piling detection device embodiment, please refer to the corresponding content in the aforementioned data piling detection method embodiment.

[0128] This invention provides a data accumulation detection system, such as... Figure 5 As shown, the system includes:

[0129] The cross-network data exchange device 50, the cross-network data exchange system 51, and the aforementioned data accumulation detection device 52 are connected in sequence; wherein, the cross-network data exchange device 50 includes: multiple data transmission source ends 501, multiple boundary devices 502, and multiple data transmission destination ends 503;

[0130] Each data transmission source 501 is used to transmit the data to be transmitted to the corresponding boundary device 502;

[0131] Each boundary device 502 is used to transmit the data to be transmitted from the corresponding data transmission source 501 to the corresponding data transmission destination 503;

[0132] Each data transmission destination 503 is used to receive the data to be transmitted from the corresponding boundary device 502;

[0133] The cross-network data exchange system 51 is used to obtain the link connection status and link connection information corresponding to each data transmission source 501 based on the connection relationship between each data transmission source 501, each boundary device 502, and each data transmission destination 503, and send the link connection status and link connection information to the data accumulation detection device 52.

[0134] This invention also provides an electronic device, see [link to relevant documentation]. Figure 6 As shown, the electronic device includes a processor 130 and a memory 131. The memory 131 stores machine-executable instructions that can be executed by the processor 130. The processor 130 executes the machine-executable instructions to implement the above-described data accumulation detection method.

[0135] Furthermore, Figure 6 The electronic device shown also includes a bus 132 and a communication interface 133, with the processor 130, the communication interface 133 and the memory 131 connected via the bus 132.

[0136] The memory 131 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 133 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network. The bus 132 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0137] Processor 130 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 130 or by instructions in software form. Processor 130 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 131, and processor 130 reads the information in memory 131 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0138] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the aforementioned data accumulation detection method. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0139] The data accumulation detection method, apparatus, and electronic device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0140] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting data accumulation, characterized in that, The method includes: Based on the link connection status of each data transmission source obtained in advance, the data transmission source to be detected is determined, and a communication connection is established with the data transmission source to be detected. Based on the target type of the data transmission source to be detected, and the detection configuration information corresponding to various pre-generated types, the target detection configuration information corresponding to the target type is determined; wherein, the various types include at least: file system, message queue, and database; Based on the target detection configuration information, the target detection value of the data transmission source is obtained; Based on the target detection value and the detection thresholds corresponding to various types generated in advance, it is determined whether data accumulation occurs at the data transmission source of the data to be detected.

2. The method according to claim 1, characterized in that, The detection configuration information for each of the various types is generated in the following way: Based on the pre-acquired quantitative indicators configured for various types, detection configuration information corresponding to each type is generated.

3. The method according to claim 2, characterized in that, The step of generating detection configuration information for each type based on pre-acquired quantitative indicators configured for various types includes: Obtain the first quantitative indicator of the user's file system configuration; wherein, the first quantitative indicator includes: the number of remaining files and the total size of remaining files; Based on each of the first quantitative indicators, generate the first detection configuration information corresponding to the file system; Obtain the second quantitative metric configured by the user for the message queue; wherein the second quantitative metric includes: topic name and remaining consumption offset; Based on each of the second quantification indicators, generate the second detection configuration information corresponding to the message queue; Obtain the third quantitative indicator of the user's database configuration; wherein, the third quantitative indicator includes: the number of remaining rows of records to be processed and the total size of remaining records to be processed; Based on each of the third quantitative indicators, the corresponding third detection configuration information for the database is generated.

4. The method according to claim 3, characterized in that, Based on the target detection configuration information, the steps for obtaining the target detection value at the source end of the data transmission to be detected include: Based on the target detection configuration information and the preset time period, the target detection value of the data transmission source corresponding to each time period is obtained.

5. The method according to claim 1, characterized in that, The link connection status includes: connected status and disconnected status; the steps of determining the data transmission source to be detected based on the pre-acquired link connection status of each data transmission source and establishing a communication connection with the data transmission source to be detected include: Based on the link connection status corresponding to each data transmission source end obtained in advance, determine the target data transmission source end corresponding to the connected status; The target data transmission source is identified as the data transmission source to be detected, and a communication connection is established with the data transmission source to be detected.

6. The method according to claim 4, characterized in that, The detection thresholds for each type include: the number of times corresponding to the time period, and the index threshold corresponding to the quantification index; wherein, the index threshold includes at least one of the following: index quantity threshold and index size threshold; The step of determining whether data accumulation occurs at the data transmission source end to be detected, based on the target detection value and the pre-generated detection thresholds corresponding to various types, includes: Based on the target type of the data transmission source corresponding to the target detection value, a target detection threshold corresponding to the target type is determined; wherein, the target detection threshold includes: a target count threshold and a target index threshold; The target detection value is compared with the target index threshold to obtain the comparison result; If the comparison result indicates that the target detection value is greater than the target index threshold, obtain the first target detection value corresponding to the next time period; if the first target detection value is greater than the target detection value, record the number of times corresponding to the time period as 1. The cumulative number of times corresponding to the time period is counted. If the number of times corresponding to the time period is not greater than the target number threshold, the first target detection value is used as the new target detection value. The step of comparing the target detection value with the target index threshold to obtain the comparison result is repeated until the number of times corresponding to the time period is greater than the target number threshold. An alarm message is generated corresponding to the data transmission source to be detected; wherein the alarm message is used to indicate that data accumulation has occurred at the data transmission source to be detected.

7. The method according to claim 6, characterized in that, After generating the alarm information corresponding to the data transmission source to be detected, the following steps are performed: Based on the pre-obtained link connection information corresponding to the source end of the data transmission to be detected, the target boundary device and the target data transmission destination end corresponding to the source end of the data transmission to be detected are identified as the target abnormal device to be detected.

8. A data accumulation detection device, characterized in that, The device includes: The first determining module is used to determine the data transmission source to be detected based on the link connection status corresponding to each data transmission source in advance, and to establish a communication connection with the data transmission source to be detected. The second determining module is used to determine the target detection configuration information corresponding to the target type based on the target type of the data transmission source end to be detected and the detection configuration information corresponding to the various types generated in advance; wherein, the various types include at least: file system, message queue, and database; The acquisition module is used to acquire the target detection value at the source end of the data transmission to be detected based on the target detection configuration information. The judgment module is used to determine whether data accumulation occurs at the data transmission source of the data to be detected, based on the target detection value and the detection thresholds corresponding to various types generated in advance.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data accumulation detection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the data accumulation detection method according to any one of claims 1-7.

Images