Methods, devices, equipment, and storage media for end-to-end monitoring of massive amounts of data.

Abstract

This invention relates to artificial intelligence technology and discloses a method for end-to-end monitoring of massive financial data. The method includes: collecting link data, log data, and indicator data; selecting a message queue sharding strategy based on data volume balance to divide the link data, log data, and indicator data into sharded link data, sharded log data, and sharded indicator data; storing the standardized link data and standardized log data in a first database with a hot-warm architecture; extracting associated indicator data related to the sharded indicator data from the standardized link data and standardized log data; performing secondary aggregation on the associated indicator data and sharded indicator data to obtain aggregated indicator data, and storing the standardized link data, standardized log data, and aggregated indicator data in the hot-warm architecture database. This invention also proposes a massive data end-to-end monitoring device, electronic device, and storage medium. This invention can improve the accuracy and maintainability of massive data monitoring.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and computer-readable storage medium for full-link monitoring of massive data. Background Technology

[0002] With the explosive growth of enterprise data volume and the transformation of enterprises to microservice technology architecture, services are split according to different dimensions. A single request often involves multiple services. In particular, financial and medical services are usually developed by different teams, deployed on different servers, and even span different data centers. It is necessary to monitor the status of data (metrics, log records, and tracking data) throughout the entire chain to ensure the normal operation of the system.

[0003] Currently, full-link monitoring of massive data typically employs two methods: full sampling and non-full sampling. Full sampling, due to the sheer volume of data, uses a single sampling method, resulting in insufficient processing capacity, leading to system crashes and data loss. This necessitates the use of low sampling rates, resulting in insufficient sampled data and inaccurate anomaly monitoring. Non-full sampling, on the other hand, generates inaccurate metrics, failing to provide correct data indicators for decision-making (such as alerts). Furthermore, during troubleshooting, the lack of tracking data prevents the resolution of monitoring issues based on the monitoring results.

[0004] In summary, current methods for monitoring massive amounts of data across the entire data chain suffer from low accuracy and poor maintainability. Summary of the Invention

[0005] This invention provides a method, device, and computer-readable storage medium for full-link monitoring of massive data, with the main purpose of solving the problems of low accuracy and maintainability in monitoring massive data.

[0006] To achieve the above objectives, the present invention provides a method for end-to-end monitoring of massive data, comprising:

[0007] Collect link data, log data, and indicator data;

[0008] The data volume of the link data, the log data, and the indicator data is statistically analyzed. A message queue sharding strategy is selected based on the data volume balance. According to the message queue sharding strategy, the link data, the log data, and the indicator data are divided into sharded link data, sharded log data, and sharded indicator data.

[0009] The fragmented link data and the fragmented log data are parsed into regularized link data and regularized log data in a preset format. The regularized link data and the regularized log data are stored in the first database of the hot and warm architecture, and the regularized link data and the regularized log data are displayed according to the first preset dimension.

[0010] Extract the associated indicator data related to the sharding indicator data from the regularized link data and the regularized log data;

[0011] The associated indicator data and the segmented indicator data are aggregated in a secondary manner to obtain aggregated indicator data, and the aggregated indicator data is stored in the second database of the thermal architecture.

[0012] The aggregated indicator data that meets the preset indicator threshold will be pushed to the user as an alarm message, and the aggregated indicator data will be displayed according to the second preset dimension.

[0013] Optionally, the step of selecting a message queue sharding strategy based on the data volume balance includes:

[0014] When the data volume is less than a first preset threshold, message queue shards are randomly selected based on the IDs of the link data, the log data, and the indicator data.

[0015] When the data volume is greater than or equal to the first preset volume threshold and less than the second preset volume threshold, a single message queue cluster is selected, and the IP addresses of the link data, the log data and the indicator data are hashed to obtain the IP address hash value. Based on the IP address hash value, message queue shards are allocated to the single message queue cluster.

[0016] When the data volume is greater than or equal to the second preset threshold, multiple message queue clusters are selected, and message queue shards are allocated to the multiple message queue clusters according to the IP address hash value.

[0017] Optionally, parsing the fragmented link data and the fragmented log data into regularized link data and regularized log data in a preset format includes:

[0018] Data cleaning is performed on the fragmented link data and the fragmented log data to obtain target fragmented link data and target fragmented log data;

[0019] The target fragment link data and the target fragment log data are parsed into regularized link data and regularized log data with the same format.

[0020] Optionally, the step of performing secondary aggregation on the associated indicator data and the sharded indicator data to obtain aggregated indicator data includes:

[0021] Analyze the source and application of the correlation index data and the segmentation index data;

[0022] Based on the source application and the IP address, the associated indicator data and the sharded indicator data are aggregated for the first time from four dimensions: source application, IP address, indicator name and time, to obtain the first aggregated indicator data;

[0023] Based on the source application, the first aggregated indicator data is aggregated a second time from three dimensions: the source application, the indicator name, and time, to obtain aggregated indicator data.

[0024] Optionally, the step of extracting associated indicator data related to the sharding indicator data from the regularized link data and the regularized log data includes:

[0025] The regularized link data and the regularized log data are segmented and quantized to obtain a link data vector sequence and a log data vector sequence;

[0026] Extract keywords from the segmentation index data;

[0027] Calculate the similarity between the keyword and the word vectors in the link data vector sequence and the log data vector sequence, respectively;

[0028] Word vectors whose similarity meets a preset similarity threshold are used as association index data.

[0029] Optionally, storing the aggregated index data in the second database of the thermal architecture includes:

[0030] The aggregated index data is stored as hot data in the hot data node of the second database;

[0031] Detect the retention time of the aggregated indicator data in the second database;

[0032] The aggregated index data that has been retained for a duration longer than a preset duration is regarded as cold data and automatically migrated to the cold data node in the second database.

[0033] Optionally, displaying the regularized link data and the regularized log data according to the first preset dimension includes:

[0034] The regularized link data and the regularized log data are classified according to the ID, business category and time range respectively to obtain the ID dataset, business category dataset and time range dataset.

[0035] When query data is received, the query dimensions of the query data are analyzed, and one of the following datasets is selected based on the query dimensions: the ID dataset, the business category dataset, and the time range dataset.

[0036] Based on the query data, query and display the query results from the selected dataset.

[0037] To address the aforementioned problems, the present invention also provides a massive data end-to-end monitoring device, the device comprising:

[0038] The data acquisition module is used to collect link data, log data, and indicator data;

[0039] The balanced sharding module is used to count the data volume of the link data, the log data, and the indicator data, select a message queue sharding strategy based on the data volume, and divide the link data, the log data, and the indicator data into sharded link data, sharded log data, and sharded indicator data according to the message queue sharding strategy.

[0040] The parsing module is used to parse the fragmented link data and the fragmented log data into regularized link data and regularized log data in a preset format, store the regularized link data and the regularized log data in the first database of the hot and warm architecture, and display the regularized link data and the regularized log data according to a first preset dimension.

[0041] The data aggregation module is used to extract associated indicator data related to the sharding indicator data from the regularized link data and the regularized log data; perform secondary aggregation on the associated indicator data and the sharding indicator data to obtain aggregated indicator data, and store the aggregated indicator data in the second database of the thermal architecture;

[0042] The information alarm module is used to push the aggregated indicator data that meets the preset indicator threshold as alarm information to the user, and to display the aggregated indicator data according to the second preset dimension.

[0043] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0044] At least one processor; and,

[0045] A memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the massive data end-to-end monitoring method described above.

[0047] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned method for full-link monitoring of massive data.

[0048] This invention, through its embodiments, selects a message queue sharding strategy based on the balanced data volume. According to this strategy, the link data, log data, and indicator data are divided into sharded link data, sharded log data, and sharded indicator data. This results in more balanced data, stronger processing capabilities, avoids data loss due to data congestion, and improves the monitoring accuracy of massive amounts of data. Furthermore, it extracts associated indicator data related to the sharded indicator data from the standardized link data and log data, and performs secondary aggregation on the associated indicator data and sharded indicator data to obtain aggregated indicator data, which can be adapted to various streaming frameworks. The pipeline processing mechanism employs a two-level aggregation method to obtain instance-level and application-level data. Instance-level and application-level data are independent of each other, facilitating rapid troubleshooting when problems arise and improving the maintainability of massive amounts of data. Finally, aggregated indicator data meeting preset thresholds are pushed to users as alarm information. The standardized link data, standardized log data, and aggregated indicator data are stored in a hot-and-warm architecture database, effectively processing written data and enabling rapid querying. Simultaneously, data can be retained for a longer period while saving costs, further enhancing the maintainability of massive amounts of data. Therefore, the massive data end-to-end monitoring method, device, electronic equipment, and computer-readable storage medium proposed in this invention can solve the problems of low accuracy and poor maintainability in massive data end-to-end monitoring. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating a method for monitoring the entire chain of massive data according to an embodiment of the present invention.

[0050] Figure 2 for Figure 1 The diagram shows a detailed implementation process for one step in the massive data end-to-end monitoring method.

[0051] Figure 3 for Figure 1 The diagram shows a detailed implementation process for another step in the massive data end-to-end monitoring method.

[0052] Figure 4 This is a functional block diagram of a massive data end-to-end monitoring device provided in an embodiment of the present invention;

[0053] Figure 5This is a schematic diagram of the structure of an electronic device for implementing the massive data end-to-end monitoring method according to an embodiment of the present invention.

[0054] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0056] This application provides a method for monitoring the entire chain of massive data. The executing entity of the massive data full-link monitoring method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application embodiment: a server, a terminal, etc. In other words, the massive data full-link monitoring method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0057] Reference Figure 1 The diagram shown is a flowchart illustrating a method for monitoring the entire chain of massive data according to an embodiment of the present invention. In this embodiment, the method for monitoring the entire chain of massive data includes:

[0058] S1. Collect link data, log data, and indicator data.

[0059] In this embodiment of the invention, the trace data, log data, and metrics data can be collected using an Agent.

[0060] In this embodiment of the invention, the link data, the log data, and the indicator data are link data, log data, and indicator data generated by different financial or medical services, such as link data, log data, and indicator data generated by services corresponding to a trading system.

[0061] In this embodiment of the invention, each piece of data in the link data, log data, and indicator data has a corresponding ID and IP address.

[0062] S2. Calculate the data volume of the link data, the log data, and the indicator data. Select a message queue sharding strategy based on the data volume and divide the link data, the log data, and the indicator data into sharded link data, sharded log data, and sharded indicator data according to the message queue sharding strategy.

[0063] In detail, the message queue sharding strategy selected based on the data volume balance described in S2 includes:

[0064] When the data volume is less than a first preset threshold, message queue shards are randomly selected based on the IDs of the link data, the log data, and the indicator data.

[0065] When the data volume is greater than or equal to the first preset volume threshold and less than the second preset volume threshold, a single message queue cluster is selected, and the IP addresses of the link data, the log data and the indicator data are hashed to obtain the IP address hash value. Based on the IP address hash value, message queue fragments are allocated to the single message queue cluster.

[0066] When the data volume is greater than or equal to the second preset threshold, multiple message queue clusters are selected, and message queue shards are allocated to the multiple message queue clusters according to the IP address hash value.

[0067] In this embodiment of the invention, a lock-free circular queue is used to cache the link data, the log data, and the indicator data. Then, an independent thread consumes the lock-free circular queue and reports the link data, the log data, and the indicator data to a message queue.

[0068] In this embodiment of the invention, a message queue sharding strategy is selected based on the balanced data volume. For example, when the data volume in Skywalking is less than 500 million segments / day, message queue shards are randomly selected based on the IDs of the link data, log data, and metric data. When the data volume in Skywalking is greater than or equal to 500 million segments / day but less than 10 billion segments / day, a single message queue cluster is selected, and the IP addresses of the link data, log data, and metric data are hashed to obtain IP address hash values. Message queue shards are then allocated to the single message queue cluster based on the IP address hash values. When the data volume in Skywalking is greater than or equal to 10 billion segments / day, multiple message queue clusters are selected, and message queue shards are allocated to the multiple message queue clusters based on the IP address hash values.

[0069] In this embodiment of the invention, when the data volume is less than a first preset threshold, the link data, the log data, and the indicator data are placed into a message queue fragment corresponding to the ID to obtain fragmented link data, fragmented log data, and fragmented indicator data; when the data volume is greater than or equal to the first preset threshold, the link data, the log data, and the indicator data are placed into a message queue fragment corresponding to the IP address hash value to obtain fragmented link data, fragmented log data, and fragmented indicator data.

[0070] In this embodiment of the invention, a message queue sharding strategy is selected based on the data volume balance. According to the message queue sharding strategy, the link data, the log data, and the indicator data are divided into sharded link data, sharded log data, and sharded indicator data, which makes the data more balanced, has stronger processing capabilities, avoids data loss due to data compression, and improves the monitoring accuracy of massive data.

[0071] S3. Parse the sharded link data and the sharded log data into regularized link data and regularized log data in a preset format, store the regularized link data and the regularized log data in the first database of the hot-warm architecture, and display the regularized link data and the regularized log data according to the first preset dimension.

[0072] In this embodiment of the invention, the first database of the hot-warm architecture is a database containing hot data nodes and warm data nodes. Typically, hot data nodes store the hot data that users care about most, while warm data nodes store cold or warm data that users have lower priority. This can be used to store log-type databases.

[0073] Specifically, S3, parsing the fragmented link data and the fragmented log data into regularized link data and regularized log data in a preset format, includes:

[0074] Data cleaning is performed on the fragmented link data and the fragmented log data to obtain target fragmented link data and target fragmented log data;

[0075] The target fragment link data and the target fragment log data are parsed into regularized link data and regularized log data with the same format.

[0076] In this embodiment of the invention, a policy mechanism can be used to clean the sharded link data and the sharded log data, removing data that is not within the expected time window.

[0077] In this embodiment of the invention, the regularized link data and the regularized log data of nearly 2 days can be stored as hot data in the hot data nodes of the first database, and the regularized link data and the regularized log data of more than 2 days can be automatically migrated to the cold data nodes of the first database as cold data.

[0078] In this embodiment of the invention, a hot-warm architecture is adopted for storage according to the different read and write requirements of the regularized link data and the regularized log data. This architecture can effectively process the written data and can be used for fast querying. At the same time, it can retain the data for a longer period of time while saving costs.

[0079] Furthermore, the description in S3 of displaying the regularized link data and the regularized log data according to the first preset dimension includes:

[0080] The regularized link data and the regularized log data are classified according to the ID, business category and time range respectively to obtain the ID dataset, business category dataset and time range dataset.

[0081] When query data is received, the query dimensions of the query data are analyzed, and one of the following datasets is selected based on the query dimensions: the ID dataset, the business category dataset, and the time range dataset.

[0082] Based on the query data, query and display the query results from the selected dataset.

[0083] In this embodiment of the invention, precise queries can be performed by [ID of log data and indicator data], precise queries can be performed by [business category], and fuzzy queries can be performed by [time range + keyword].

[0084] S4. Extract the associated indicator data related to the sharding indicator data from the regularized link data and the regularized log data.

[0085] Specifically, S4 includes:

[0086] The regularized link data and the regularized log data are segmented and quantized to obtain a link data vector sequence and a log data vector sequence;

[0087] Extract keywords from the segmentation index data;

[0088] Calculate the similarity between the keyword and the word vectors in the link data vector sequence and the log data vector sequence, respectively;

[0089] Word vectors whose similarity meets a preset similarity threshold are used as association index data.

[0090] In this embodiment of the invention, keywords can be extracted using algorithms such as TF-IDF (term frequency–inverse document frequency), TextRank keyword extraction algorithm, and K-means clustering algorithm.

[0091] In this embodiment of the invention, algorithms such as cosine similarity, Jaccard similarity coefficient, and Pearson correlation coefficient can be used to calculate the similarity between the keywords and the word vectors in the link data vector sequence and the log data vector sequence.

[0092] In this embodiment of the invention, associated indicator data related to the sharded indicator data is extracted from the regularized link data and the regularized log data, and the link data and log data are linked with the indicator data, which is beneficial for troubleshooting and improves the maintainability of full-link monitoring of massive data.

[0093] S5. Perform secondary aggregation on the associated indicator data and the segmented indicator data to obtain aggregated indicator data, and store the aggregated indicator data in the second database of the thermal architecture.

[0094] For details, please refer to Figure 2 As shown, the secondary aggregation of the associated indicator data and the segmented indicator data in S5 to obtain aggregated indicator data includes:

[0095] S51. Analyze the source and application of the correlation index data and the segmentation index data;

[0096] S52. Based on the source application and the IP address, the associated indicator data and the sharded indicator data are aggregated for the first time from four dimensions: the source application, the IP address, the indicator name, and the time, to obtain the first aggregated indicator data.

[0097] S53. Based on the source application, the first aggregated indicator data is aggregated a second time from three dimensions: the source application, the indicator name, and time, to obtain aggregated indicator data.

[0098] In this embodiment of the invention, the associated indicator data and the sharded indicator data include the application from which the data originates, the IP address of the data, and the indicator data.

[0099] In this embodiment of the invention, based on the source application and the IP address, the associated indicator data and the shard indicator data of the same application and the same IP address are aggregated for the first time from four dimensions: source application, IP address, indicator name, and time, to obtain instance-level first aggregated indicator data. The time dimension includes time scales such as minutes, hours, and days. Further, the first aggregated indicator data of the same source application are aggregated for the second time from three dimensions: source application, indicator name, and time, to obtain application-level aggregated indicator data.

[0100] In this embodiment of the invention, the associated indicator data and the sharded indicator data are aggregated in a two-level manner to obtain aggregated indicator data. This can be adapted to various streaming architecture pipe-line processing mechanisms. The two-level aggregation method yields instance-level and application-level data, which are independent of each other, facilitating quick troubleshooting of the source of problems when they occur.

[0101] Further, see Figure 3 As shown, in step S5, storing the aggregated index data in the second database of the thermal architecture includes:

[0102] S54. Store the aggregated index data as hot data in the hot data node of the second database;

[0103] S55. Detect the retention time of the aggregated indicator data in the second database;

[0104] S56. The aggregated index data that has been retained for a duration longer than a preset duration is regarded as cold data and automatically migrated to the cold data node in the second database.

[0105] In this embodiment of the invention, the aggregated index data for nearly 2 days can be stored as hot data in the hot data node of the second database. Because hot data is read and written frequently and has high performance requirements, SSDs can be used to build the hot data node. The aggregated index data for more than 2 days can be stored as cold data and automatically migrated to the cold data node of the second database. Because cold data has a larger storage density and needs to be retained for a longer period of time, mechanical hard drives can be used to build the cold data node.

[0106] In this embodiment of the invention, a hot-warm architecture is used for storage according to the different read and write requirements of the aggregated index data. This architecture can effectively process the written data and can be used for fast querying. At the same time, it can retain the data for a longer period of time while saving costs.

[0107] S6. Push the aggregated indicator data that meets the preset indicator threshold as alarm information to the user, and display the aggregated indicator data according to the second preset dimension.

[0108] In this embodiment of the invention, when the aggregated indicator data reaches a preset indicator threshold, it indicates that the aggregated indicator data is abnormal, and the abnormal aggregated indicator data is pushed to the user as an alarm message.

[0109] In this embodiment of the invention, the second preset dimension can be defined and displayed according to query requirements or user requirements.

[0110] This invention, through its embodiments, selects a message queue sharding strategy based on the balanced data volume. According to this strategy, the link data, log data, and indicator data are divided into sharded link data, sharded log data, and sharded indicator data. This results in more balanced data, stronger processing capabilities, avoids data loss due to data congestion, and improves the monitoring accuracy of massive amounts of data. Furthermore, it extracts associated indicator data related to the sharded indicator data from the standardized link data and log data, and performs secondary aggregation on the associated indicator data and sharded indicator data to obtain aggregated indicator data, which can be adapted to various streaming frameworks. The pipeline processing mechanism employs a two-level aggregation method to obtain instance-level and application-level data. These two levels of data are independent, facilitating rapid troubleshooting of problems and improving the maintainability of massive datasets. Finally, aggregated indicator data meeting preset thresholds are pushed as alarm information to users. The standardized pipeline data, standardized log data, and aggregated indicator data are stored in a hot-and-warm architecture database, effectively processing written data and enabling rapid querying. Simultaneously, data can be retained for a longer period while saving costs, further enhancing the maintainability of massive datasets. Therefore, the massive data end-to-end monitoring method proposed in this invention can solve the problems of low accuracy and poor maintainability in massive data end-to-end monitoring.

[0111] like Figure 4 The diagram shown is a functional block diagram of a massive data end-to-end monitoring device provided in an embodiment of the present invention.

[0112] The massive data end-to-end monitoring device 100 of this invention can be installed in an electronic device. Depending on the functions implemented, the massive data end-to-end monitoring device 100 may include a data acquisition module 101, a balanced data sharding module 102, a parsing module 103, a data aggregation module 104, and an information alarm module 105. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0113] In this embodiment, the functions of each module / unit are as follows:

[0114] The data acquisition module 101 is used to collect link data, log data and indicator data;

[0115] The balanced sharding module 102 is used to count the data volume of the link data, the log data and the indicator data, select a message queue sharding strategy based on the data volume, and divide the link data, the log data and the indicator data into sharded link data, sharded log data and sharded indicator data according to the message queue sharding strategy.

[0116] The parsing module 103 is used to parse the fragmented link data and the fragmented log data into regularized link data and regularized log data in a preset format, store the regularized link data and the regularized log data in the first database of the hot and warm architecture, and display the regularized link data and the regularized log data according to a first preset dimension.

[0117] The data aggregation module 104 is used to extract associated indicator data related to the sharding indicator data from the regularized link data and the regularized log data; perform secondary aggregation on the associated indicator data and the sharding indicator data to obtain aggregated indicator data, and store the aggregated indicator data in the second database of the thermal architecture;

[0118] The information alarm module 105 is used to push the aggregated indicator data that meets the preset indicator threshold as alarm information to the user, and to display the aggregated indicator data according to the second preset dimension.

[0119] In detail, the modules in the massive data end-to-end monitoring device 100 described in this embodiment of the invention employ the same methods as described above. Figures 1 to 3 The method uses the same techniques as the full-link monitoring method for massive data described above and can produce the same technical effects, so it will not be elaborated here.

[0120] like Figure 5 The diagram shown is a structural schematic of an electronic device for implementing a method for full-link monitoring of massive data, according to an embodiment of the present invention.

[0121] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a massive data full-link monitoring program.

[0122] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a massive data end-to-end monitoring program) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0123] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a massive data end-to-end monitoring program, but also to temporarily store data that has been output or will be output.

[0124] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0125] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0126] Figure 5 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 5 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0127] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0128] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0129] The massive data end-to-end monitoring program stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0130] Collect link data, log data, and indicator data;

[0131] The data volume of the link data, the log data, and the indicator data is statistically analyzed. A message queue sharding strategy is selected based on the data volume balance. According to the message queue sharding strategy, the link data, the log data, and the indicator data are divided into sharded link data, sharded log data, and sharded indicator data.

[0132] The fragmented link data and the fragmented log data are parsed into regularized link data and regularized log data in a preset format. The regularized link data and the regularized log data are stored in the first database of the hot and warm architecture, and the regularized link data and the regularized log data are displayed according to the first preset dimension.

[0133] Extract the associated indicator data related to the sharding indicator data from the regularized link data and the regularized log data;

[0134] The associated indicator data and the segmented indicator data are aggregated in a secondary manner to obtain aggregated indicator data, and the aggregated indicator data is stored in the second database of the thermal architecture.

[0135] The aggregated indicator data that meets the preset indicator threshold will be pushed to the user as an alarm message, and the aggregated indicator data will be displayed according to the second preset dimension.

[0136] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0137] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0138] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0139] Collect link data, log data, and indicator data;

[0140] The data volume of the link data, the log data, and the indicator data is statistically analyzed. A message queue sharding strategy is selected based on the data volume balance. According to the message queue sharding strategy, the link data, the log data, and the indicator data are divided into sharded link data, sharded log data, and sharded indicator data.

[0141] The fragmented link data and the fragmented log data are parsed into regularized link data and regularized log data in a preset format. The regularized link data and the regularized log data are stored in the first database of the hot and warm architecture, and the regularized link data and the regularized log data are displayed according to the first preset dimension.

[0142] Extract the associated indicator data related to the sharding indicator data from the regularized link data and the regularized log data;

[0143] The associated indicator data and the segmented indicator data are aggregated in a secondary manner to obtain aggregated indicator data, and the aggregated indicator data is stored in the second database of the thermal architecture.

[0144] The aggregated indicator data that meets the preset indicator threshold will be pushed to the user as an alarm message, and the aggregated indicator data will be displayed according to the second preset dimension.

[0145] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0146] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0148] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0149] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0150] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0151] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0152] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for end-to-end monitoring of massive data, characterized in that, The method includes: Collect link data, log data, and indicator data; The data volume of the link data, the log data, and the indicator data is statistically analyzed. A message queue sharding strategy is selected based on the data volume balance. According to the message queue sharding strategy, the link data, the log data, and the indicator data are divided into sharded link data, sharded log data, and sharded indicator data. The fragmented link data and the fragmented log data are parsed into regularized link data and regularized log data in a preset format. The regularized link data and the regularized log data are stored in the first database of the hot and warm architecture, and the regularized link data and the regularized log data are displayed according to the first preset dimension. Extract the associated indicator data related to the sharding indicator data from the regularized link data and the regularized log data; The associated indicator data and the segmented indicator data are aggregated in a secondary manner to obtain aggregated indicator data, and the aggregated indicator data is stored in the second database of the thermal architecture. The aggregated indicator data that meets the preset indicator threshold will be pushed to the user as alarm information, and the aggregated indicator data will be displayed according to the second preset dimension. The step of extracting associated indicator data related to the sharding indicator data from the regularized link data and the regularized log data includes: The regularized link data and the regularized log data are segmented and quantized to obtain a link data vector sequence and a log data vector sequence; Extract keywords from the segmentation index data; Calculate the similarity between the keyword and the word vectors in the link data vector sequence and the log data vector sequence, respectively; Word vectors whose similarity meets a preset similarity threshold are used as association index data.

2. The method for monitoring the entire chain of massive data as described in claim 1, characterized in that, The step of selecting a message queue sharding strategy based on the data volume balance includes: When the data volume is less than a first preset threshold, message queue shards are randomly selected based on the IDs of the link data, the log data, and the indicator data. When the data volume is greater than or equal to the first preset volume threshold and less than the second preset volume threshold, a single message queue cluster is selected, and the IP addresses of the link data, the log data and the indicator data are hashed to obtain the IP address hash value. Based on the IP address hash value, message queue fragments are allocated to the single message queue cluster. When the data volume is greater than or equal to the second preset threshold, multiple message queue clusters are selected, and message queue shards are allocated to the multiple message queue clusters according to the IP address hash value.

3. The method for monitoring the entire chain of massive data as described in claim 1, characterized in that, The step of parsing the fragmented link data and the fragmented log data into regularized link data and regularized log data in a preset format includes: Data cleaning is performed on the fragmented link data and the fragmented log data to obtain target fragmented link data and target fragmented log data; The target fragment link data and the target fragment log data are parsed into regularized link data and regularized log data with the same format.

4. The method for monitoring the entire chain of massive data as described in claim 1, characterized in that, The step of performing secondary aggregation on the associated indicator data and the segmented indicator data to obtain aggregated indicator data includes: Analyze the source and application of the correlation index data and the segmentation index data; Based on the IP addresses of the source application, the associated indicator data, and the shard indicator data, the associated indicator data and the shard indicator data are aggregated for the first time from four dimensions: the source application, the IP address, the indicator name, and the time, to obtain the first aggregated indicator data. Based on the source application, the first aggregated indicator data is aggregated a second time from three dimensions: the source application, the indicator name, and time, to obtain aggregated indicator data.

5. The method for full-link monitoring of massive data as described in claim 1, characterized in that, The step of storing the aggregated index data in the second database of the thermal architecture includes: The aggregated index data is stored as hot data in the hot data node of the second database; Detect the retention time of the aggregated indicator data in the second database; The aggregated index data that has been retained for a duration longer than a preset duration is regarded as cold data and automatically migrated to the cold data node in the second database.

6. The method for full-link monitoring of massive data as described in claim 1, characterized in that, The step of displaying the regularized link data and the regularized log data according to the first preset dimension includes: The regularized link data and the regularized log data are classified according to the ID, business category and time range of the regularized link data and the regularized log data, respectively, to obtain ID dataset, business category dataset and time range dataset; When query data is received, the query dimensions of the query data are analyzed, and one of the following datasets is selected based on the query dimensions: the ID dataset, the business category dataset, and the time range dataset. Based on the query data, query and display the query results from the selected dataset.

7. A massive data end-to-end monitoring device, used to implement the massive data end-to-end monitoring method as described in any one of claims 1 to 6, characterized in that, The device includes: The data acquisition module is used to collect link data, log data, and indicator data; The balanced sharding module is used to count the data volume of the link data, the log data, and the indicator data, select a message queue sharding strategy based on the data volume, and divide the link data, the log data, and the indicator data into sharded link data, sharded log data, and sharded indicator data according to the message queue sharding strategy. The parsing module is used to parse the fragmented link data and the fragmented log data into regularized link data and regularized log data in a preset format, store the regularized link data and the regularized log data in the first database of the hot and warm architecture, and display the regularized link data and the regularized log data according to a first preset dimension. The data aggregation module is used to extract associated indicator data related to the sharding indicator data from the regularized link data and the regularized log data; perform secondary aggregation on the associated indicator data and the sharding indicator data to obtain aggregated indicator data, and store the aggregated indicator data in the second database of the thermal architecture; The information alarm module is used to push the aggregated indicator data that meets the preset indicator threshold as alarm information to the user, and to display the aggregated indicator data according to the second preset dimension.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the massive data end-to-end monitoring method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the massive data full-link monitoring method as described in any one of claims 1 to 6.

Images