Data processing method, system, device, computer equipment and storage medium

Abstract

The present application relates to a data processing method, system, device, computer equipment, storage medium and computer program product, and relates to the field of cloud computing technology. The method includes: obtaining unitized monitoring data through an acquisition terminal, and determining the unitized indicator type of the unitized monitoring data according to a preset unitized indicator classification rule; based on a preset sending strategy, sending unitized monitoring information corresponding to the unitized monitoring data through a gateway sending node; based on a preset receiving strategy, receiving the unitized monitoring information through a gateway receiving node; determining the target storage space of the unitized monitoring data according to the type identification of the unitized monitoring data and a preset storage rule through a data processing node, and performing storage processing on the unitized monitoring data; the unitized monitoring data is used to monitor the unitized system. The present method can realize data processing of the unitized monitoring system and complete the monitoring of the unitized system.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data processing method, system, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] A unit is a self-contained collection capable of completing all business operations. This collection includes all services required for all businesses, as well as the data assigned to the unit. A unitized architecture uses units as the fundamental unit of system deployment, deploying multiple units across all data centers. Each unit hosts all the applications required by the system, while data is a subset of the full data set, partitioned horizontally according to a specific dimension (usually the customer dimension). Unitization is the process of designing and refining services and data to conform to the characteristics of a unit.

[0003] However, there is currently a lack of data processing methods suitable for monitoring unitized systems. Summary of the Invention

[0004] Based on this, it is necessary to provide a data processing method, system, device, computer equipment, computer-readable storage medium and computer program product that can be applied to the monitoring of unitized systems in order to address the above technical problems.

[0005] In a first aspect, the present application provides a data processing method. The method is applied to a unitized monitoring system, wherein the unitized monitoring system includes a collection terminal, a gateway sending node, a gateway receiving node, and a data processing node; the method includes:

[0006] Acquire unitized monitoring data through the acquisition end, and determine the unitized indicator type of the unitized monitoring data according to a preset unitized indicator classification rule; the unitized monitoring data includes unitized metadata and monitoring data corresponding to the unitized metadata;

[0007] Based on a preset sending strategy, the unitized monitoring information corresponding to the unitized monitoring data is sent through the gateway sending node; the unitized monitoring information includes the unitized monitoring data and a type identifier; the type identifier is used to indicate the unitized indicator type of the unitized monitoring data;

[0008] Based on a preset receiving strategy, receiving the unitized monitoring information through the gateway receiving node;

[0009] The data processing node determines the target storage space of the unitized monitoring data according to the type identification of the unitized monitoring data and the preset storage rules, and performs storage processing on the unitized monitoring data; the unitized monitoring data is used to monitor the unitized system.

[0010] In one embodiment, determining, by the data processing node, a target storage space for the unitized monitoring data according to a type identifier of the unitized monitoring data and a preset storage rule, and performing storage processing on the unitized monitoring data includes:

[0011] Based on the corresponding relationship between the preset unitized indicator type and the target storage space, the target storage space of the unitized monitoring data is determined by the data processing node, and the unitized monitoring data is stored in the target storage space.

[0012] In one embodiment, when the unitized indicator type of the unitized monitoring data is a link type, the method further includes:

[0013] Performing topological calculation on the unitized monitoring data by the data processing node to obtain a signature value corresponding to each link node included in the unitized monitoring data;

[0014] storing, by the data processing node, a target mapping relationship between the signature value and the unitized monitoring data in a first database;

[0015] Determining, by the data processing node according to the signature value corresponding to each link node, the target point edge relationship of the link included in the unitized monitoring data;

[0016] The target point-edge relationship is stored in a second database through the data processing node.

[0017] In one embodiment, the sending of the unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node based on a preset sending strategy includes:

[0018] caching the unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node;

[0019] Packaging the unitized monitoring information through the gateway sending node to obtain packaged unitized monitoring information;

[0020] The packaged unitized monitoring information is compressed by the gateway sending node to obtain sending data, and the sending data is sent.

[0021] In one embodiment, the receiving the unitized monitoring information through the gateway receiving node based on a preset receiving strategy includes:

[0022] consuming the sent data via the gateway receiving node;

[0023] decompressing the transmission data through the gateway receiving node to obtain decompressed transmission data;

[0024] The decompressed sending data is unpacked by the gateway receiving node to obtain the unitized monitoring information.

[0025] In one embodiment, the sending the data includes:

[0026] Sending the sending data to the message middleware;

[0027] The consuming of the sent data by the gateway receiving node includes:

[0028] The sending data is consumed from the message middleware through the gateway receiving node.

[0029] In a second aspect, the present application further provides a data processing system. The system comprises:

[0030] The collection end is used to obtain unitized monitoring data and determine the unitized indicator type of the unitized monitoring data according to a preset unitized indicator classification rule; the unitized monitoring data includes unitized metadata and monitoring data corresponding to the unitized metadata;

[0031] The gateway sending node is used to send the unitized monitoring information corresponding to the unitized monitoring data based on a preset sending strategy; the unitized monitoring information includes the unitized monitoring data and a type identifier; the type identifier is used to indicate the unitized indicator type of the unitized monitoring data;

[0032] A gateway receiving node, configured to receive the unitized monitoring information based on a preset receiving strategy;

[0033] A data processing node is used to determine the target storage space of the unitized monitoring data according to the type identification and preset storage rules of the unitized monitoring data, and to store and process the unitized monitoring data; the unitized monitoring data is used to monitor the unitized system.

[0034] In one embodiment, the data processing node is specifically configured to:

[0035] Based on a preset correspondence between the unitized indicator type and the target storage space, the target storage space of the unitized monitoring data is determined, and the unitized monitoring data is stored in the target storage space.

[0036] In one embodiment, when the unitized indicator type of the unitized monitoring data is a link type, the data processing node is further configured to:

[0037] Performing topological calculation on the unitized monitoring data to obtain a signature value corresponding to each link node contained in the unitized monitoring data;

[0038] Storing a target mapping relationship between the signature value and the unitized monitoring data in a first database;

[0039] Determining the target point-edge relationship of the link included in the unitized monitoring data according to the signature value corresponding to each link node;

[0040] The target point-edge relationship is stored in a second database.

[0041] In one embodiment, the gateway sending node is specifically configured to:

[0042] caching unitized monitoring information corresponding to the unitized monitoring data;

[0043] Packaging the unitized monitoring information to obtain packaged unitized monitoring information;

[0044] The packaged unitized monitoring information is compressed to obtain sending data, and the sending data is sent.

[0045] In one embodiment, the gateway receiving node is specifically configured to:

[0046] Consume the sent data;

[0047] Decompressing the transmission data to obtain decompressed transmission data;

[0048] The decompressed sending data is unpacked to obtain the unitized monitoring information.

[0049] In one embodiment, the gateway sending node is specifically configured to:

[0050] Sending the sending data to the message middleware;

[0051] The gateway receiving node is specifically configured to:

[0052] The sent data is consumed from the message middleware.

[0053] In a third aspect, the present application further provides a data processing device. The device is applied to a unitized monitoring system, wherein the unitized monitoring system includes a collection terminal, a gateway sending node, a gateway receiving node, and a data processing node; the device includes:

[0054] an acquisition module, configured to acquire unitized monitoring data through the acquisition terminal and determine the unitized indicator type of the unitized monitoring data according to a preset unitized indicator classification rule; the unitized monitoring data includes unitized metadata and monitoring data corresponding to the unitized metadata;

[0055] A sending module, configured to send, based on a preset sending strategy, unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node; the unitized monitoring information includes unitized monitoring data and a type identifier; the type identifier is used to indicate a unitized indicator type of the unitized monitoring data;

[0056] A receiving module, configured to receive the unitized monitoring information through the gateway receiving node based on a preset receiving strategy;

[0057] A data processing module is used to determine the target storage space of the unitized monitoring data and store and process the unitized monitoring data according to the type identification and preset storage rules of the unitized monitoring data through the data processing node; the unitized monitoring data is used to monitor the unitized system.

[0058] In one embodiment, the data processing module is specifically configured to:

[0059] Based on the corresponding relationship between the preset unitized indicator type and the target storage space, the target storage space of the unitized monitoring data is determined by the data processing node, and the unitized monitoring data is stored in the target storage space.

[0060] In one embodiment, when the unitized indicator type of the unitized monitoring data is a link type, the apparatus further includes:

[0061] a calculation module, configured to perform topological calculation on the unitized monitoring data through the data processing node to obtain a signature value corresponding to each link node contained in the unitized monitoring data;

[0062] a first storage module, configured to store a target mapping relationship between the signature value and the unitized monitoring data in a first database via the data processing node;

[0063] a determination module, configured to determine, through the data processing node and based on the signature value corresponding to each link node, a target point-edge relationship of the link included in the unitized monitoring data;

[0064] The second storage module is used to store the target point-edge relationship in a second database through the data processing node.

[0065] In one embodiment, the sending module is specifically configured to:

[0066] caching the unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node;

[0067] Packaging the unitized monitoring information through the gateway sending node to obtain packaged unitized monitoring information;

[0068] The packaged unitized monitoring information is compressed by the gateway sending node to obtain sending data, and the sending data is sent.

[0069] In one embodiment, the receiving module is specifically configured to:

[0070] consuming the sent data via the gateway receiving node;

[0071] decompressing the transmission data through the gateway receiving node to obtain decompressed transmission data;

[0072] The decompressed sending data is unpacked by the gateway receiving node to obtain the unitized monitoring information.

[0073] In one embodiment, the sending module is specifically configured to:

[0074] Sending the sending data to the message middleware;

[0075] The receiving module is specifically configured to:

[0076] The sending data is consumed from the message middleware through the gateway receiving node.

[0077] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps described in the first aspect above when executing the computer program.

[0078] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which performs the steps described in the first aspect when executed by a processor.

[0079] In a sixth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps described in the first aspect are performed.

[0080] The above-mentioned data processing method, system, device, computer equipment, storage medium and computer program product, the method is applied to a unitized monitoring system, the unitized monitoring system includes an acquisition end, a gateway sending node, a gateway receiving node and a data processing node; the unitized monitoring data is obtained through the acquisition end, and the unitized indicator type of the unitized monitoring data is determined according to a preset unitized indicator classification rule; the unitized monitoring data includes unitized metadata and monitoring data corresponding to the unitized metadata; based on a preset sending strategy, the unitized monitoring information corresponding to the unitized monitoring data is sent through the gateway sending node; the unitized monitoring information includes unitized monitoring data and a type identifier; the type identifier is used to indicate the unitized indicator type of the unitized monitoring data; based on a preset receiving strategy, the unitized monitoring information is received through the gateway receiving node; the target storage space of the unitized monitoring data is determined by the data processing node according to the type identifier of the unitized monitoring data and a preset storage rule, and the unitized monitoring data is stored and processed; the unitized monitoring data is used to monitor the unitized system. In this way, by collecting and processing unitized monitoring data including unitized metadata, the monitoring data is given unit attributes, enabling data processing for the unitized monitoring system and completing the monitoring of the unitized system. Furthermore, by categorizing and storing unitized monitoring data in this way, the efficiency of data storage and querying can be improved, and the high availability of the unitized monitoring system can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 is a structural block diagram of a unitized monitoring system in one embodiment;

[0082] Figure 2 1 is a flow chart of a data processing method in one embodiment;

[0083] Figure 3 is a flow chart of a data processing method in another embodiment;

[0084] Figure 4 1. A schematic flow chart of steps for sending unitized monitoring information corresponding to unitized monitoring data through a gateway sending node in one embodiment;

[0085] Figure 5 1. A schematic flow chart of steps for receiving unitized monitoring information through a gateway receiving node in one embodiment;

[0086] Figure 6 is a structural block diagram of a data processing device in one embodiment;

[0087] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0089] The data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the unitized monitoring system 100 includes an acquisition terminal 102, a gateway sending node 104, a gateway receiving node 106 and a data processing node 108. The unitized monitoring system is used to monitor the unitized system. The unitized monitoring system can be implemented through a cloud computing platform. For example, the unitized monitoring system can be implemented through Platform as a Service (PaaS). The unitized system is a system that uses units as the basic units for system deployment. Any unit deploys all applications required by the system, and the data of any unit is a part of the full data after being horizontally divided according to a certain dimension (usually the customer dimension). A unit is a self-contained set that can complete all business operations. This set contains all services required for all businesses and the data allocated to this unit. The unit can be a physical unit or a logical unit. It can be understood that the unit can be a terminal or a server. The server can be implemented as an independent server or a server cluster consisting of multiple servers.

[0090] In one embodiment, Figure 2 As shown, a data processing method is provided, which is applied to Figure 1 The unitized monitoring system in the example is used to illustrate the following steps:

[0091] Step 201: Acquire unitized monitoring data through a collection terminal, and determine the unitized indicator type of the unitized monitoring data according to a preset unitized indicator classification rule.

[0092] The unitized monitoring data includes unitized metadata and monitoring data corresponding to the unitized metadata.

[0093] In the embodiment of the present application, unitized metadata is metadata that represents unit attributes. Unitized metadata may include physical metadata and logical metadata. Physical metadata is metadata that represents physical unit attributes. Logical metadata is metadata that represents logical unit attributes. A physical unit is a unit where a machine environment, such as a server, is physically located. For example, A-001 is used to represent a server deployed at unit 001 at address A. A logical unit is a unit in a logical sense that has a certain business meaning. For example, a personal account can be divided into 10 units according to the last digit of the card, and person-01 represents the unit with the last digit of the card being 1.

[0094] The collection end can predefine the format of unitized metadata. Specifically, the collection end can add a physical variable to the container. It can then set the variable value corresponding to the physical variable based on the geographic location and number of the actual deployed physical unit. Simultaneously, the collection end can add a logical variable to the system variables. It can then set the value of the logical unit corresponding to the logical variable based on the actual logical division of each application. This allows physical units to be uniformly configured through the deployment container, reducing the amount of tools required to configure each application individually and improving the efficiency of specifying the unitized metadata format.

[0095] In one example, the collection end can add a new physical variable, _PHYSICAL_UNIT, to the container. The actual physical unit is located at address A and numbered 001. The collection end can then set the value of this physical variable to A-001 based on the location and number of the actual physical unit.

[0096] In another example, the acquisition end adds a logic variable logic_unit to the system variables.

[0097] The collection end then obtains the unitized monitoring data. Specifically, the collection end searches for and reads the physical variable's value in the container variable based on the physical variable's keyword, obtaining physical metadata. Simultaneously, the collection end searches for and reads the logical variable's value in the system variable based on the logical variable's keyword, obtaining logical metadata. The collection end then obtains the monitoring data corresponding to the unitized metadata. For example, the keyword for a physical variable is _PHYSICAL_UNIT, and the keyword for a logical variable is logic_unit.

[0098] In one example, the monitoring data corresponding to the unitized metadata includes transaction data and system data. Transaction data is data related to transaction actions and depends on the occurrence of transactions. System data is data related to the operating system related to transactions and does not depend on the occurrence of transactions. When a transaction occurs, the collection end collects transaction data in real time. The collection end scans and collects system data at preset collection intervals. For example, the collection interval can be 3 seconds. In this way, for data that depends on the occurrence of transactions, the collection end collects data in real time when the transaction occurs; for data that does not depend on the occurrence of transactions, the collection end collects data at preset collection intervals. This is in line with actual conditions, ensures the acquisition of the monitoring data corresponding to the unitized metadata, avoids wasted time in data collection, and improves data collection efficiency.

[0099] At the same time, the collection end determines the unitized indicator type of the unitized monitoring data according to the preset unitized indicator classification rules.

[0100] In one embodiment, unitized indicator types may include indicator, system, and link types. Unitized monitoring data in the indicator category refers to unitized monitoring data related to transaction actions but not the transaction link, and is used to measure the success of a transaction. Unitized monitoring data in the indicator category includes, but is not limited to, transaction volume, transaction duration, transaction success, and transaction exception type. Unitized monitoring data in the system category refers to operating system data related to the transaction. Unitized monitoring data in the system category includes, but is not limited to, monitoring data of the operating system's Java Virtual Machine (JVM) and operating system (OS). Unitized monitoring data in the link category refers to unitized monitoring data of the transaction link related to the transaction action, and is used to represent the transaction link. A transaction link is a series of services involved in a transaction. For example, a payment transaction involves multiple services. Before payment, a series of services such as a protocol query service, a blacklist verification service, and a customer authentication service are called. The data from this entire link, when strung together, constitutes a completed payment transaction. Unitized monitoring data in the link category includes, but is not limited to, link links and call relationships.

[0101] Step 202: Based on a preset sending strategy, the unitized monitoring information corresponding to the unitized monitoring data is sent through the gateway sending node.

[0102] The unitized monitoring information includes unitized monitoring data and a type identifier, wherein the type identifier is used to indicate the unitized indicator type of the unitized monitoring data.

[0103] In an embodiment of the present application, the collection end can determine the type identifier of the unitized monitoring data based on the unitized indicator type of the unitized monitoring data. Then, the collection end combines the unitized monitoring data and the type identifier of the unitized monitoring data to form unitized monitoring information. The type identifier can be a tag.

[0104] In one embodiment, when the unitized indicator types include indicator class, system class and link class, the type identifier corresponding to the unitized monitoring data of the indicator class can be metrics, the type identifier corresponding to the unitized monitoring data of the system class can be system, and the type identifier corresponding to the unitized monitoring data of the link class can be trace.

[0105] Then, the gateway sending node sends the unitized monitoring information corresponding to the unitized monitoring data based on a preset sending strategy. In one example, the gateway sending node sends the unitized monitoring information corresponding to the unitized monitoring data to the gateway receiving node.

[0106] Step 203: Based on a preset receiving strategy, the unitized monitoring information is received through the gateway receiving node.

[0107] In an embodiment of the present application, the gateway receiving node receives the unitized monitoring information based on a preset receiving strategy. In one example, the gateway receiving node receives the unitized monitoring information sent by the gateway sending node.

[0108] Step 204 : The data processing node determines the target storage space for the unitized monitoring data according to the type identifier of the unitized monitoring data and the preset storage rule, and performs storage processing on the unitized monitoring data.

[0109] Among them, the unitized monitoring data is used to monitor the unitized system.

[0110] In an embodiment of the present application, a data processing node determines the target storage space for the unitized monitoring data based on the unitized monitoring data type identifier and preset storage rules. The data processing node then stores and processes the unitized monitoring data. When monitoring the unitized system, the unitized monitoring system determines whether the unitized system has experienced a fault and determines the type of fault, etc., based on the unitized monitoring data.

[0111] In the above data processing method, the unitized monitoring data is obtained through the acquisition end, and the unitized indicator type of the unitized monitoring data is determined according to the preset unitized indicator classification rules; based on the preset sending strategy, the unitized monitoring information corresponding to the unitized monitoring data is sent through the gateway sending node; based on the preset receiving strategy, the unitized monitoring information is received through the gateway receiving node; the target storage space of the unitized monitoring data is determined according to the type identification of the unitized monitoring data and the preset storage rules through the data processing node, and the unitized monitoring data is stored and processed. In this way, by collecting and processing the unitized monitoring data including the unitized metadata, the monitoring data has unit attributes, the data processing of the unitized monitoring system is realized, and the monitoring of the unitized system is completed. At the same time, such classified storage of the unitized monitoring data can not only improve the efficiency of data storage and query, but also the downtime of the database used for one type of data storage will not affect the storage and query of other data, that is, it can improve the high availability of the unitized monitoring system.

[0112] In one embodiment, the target storage space of the unitized monitoring data is determined by the data processing node according to the type identification of the unitized monitoring data and the preset storage rules, and the specific process of storing and processing the unitized monitoring data includes the following steps: based on the correspondence between the preset unitized indicator type and the target storage space, the target storage space of the unitized monitoring data is determined by the data processing node, and the unitized monitoring data is stored in the target storage space.

[0113] In an embodiment of the present application, a data processing node may pre-set a correspondence between unitized indicator types and target storage spaces. The target storage space may be, but is not limited to, a database or a table within the database. The data processing node then determines a target storage space for the unitized monitoring data based on the pre-set correspondence between the unitized indicator types and the target storage space. The data processing node then stores the unitized monitoring data in the target storage space.

[0114] In one example, the target storage space is a database. The unitized indicator type may include an indicator class, a system class, and a link class. When the unitized indicator type is an indicator class or a system class, the data processing node determines that the target storage space for the unitized monitoring data is a first-class database. When the unitized indicator type is a link class, the data processing node determines that the target storage space for the unitized monitoring data is a second-class database. The first-class database may be a time series database. The second-class database may be a file database. For example, the first-class database may be a ClickHouse database. The second-class database may be an ElasticSearch database.

[0115] In another example, the target storage space is a table in a database. The unitized indicator type may include an indicator class, a system class, and a link class. When the unitized indicator type is an indicator class, the data processing node determines that the target storage space of the unitized monitoring data is the first table in a first-class database. When the unitized indicator type is a system class, the data processing node determines that the target storage space of the unitized monitoring data is the second table in a first-class database. When the unitized indicator type is a link class, the data processing node determines that the target storage space of the unitized monitoring data is a table in a second-class database. The first table and the second table are different tables in the first-class database.

[0116] In the above data processing method, based on the preset correspondence between unitized indicator types and target storage spaces, the target storage space for the unitized monitoring data is determined by the data processing node, and the unitized monitoring data is stored in the target storage space. This categorized storage of unitized monitoring data, with unitized monitoring data of different unitized indicator types stored in different target storage spaces, not only improves the efficiency of data storage and query, but also ensures that the downtime of the target storage space used for one type of data will not affect the storage and query of other data, thereby enhancing the high availability of the unitized monitoring system.

[0117] In one embodiment, Figure 3 As shown, when the unitized indicator type of the unitized monitoring data is a link type, the method further includes the following steps:

[0118] Step 301: Perform topology calculation on the unitized monitoring data through a data processing node to obtain a signature value corresponding to each link node included in the unitized monitoring data.

[0119] In an embodiment of the present application, the data processing node performs topological calculations on the unitized monitoring data based on the unit-dimensional aggregated topological relationship diagram to obtain a signature value corresponding to each link node contained in the unitized monitoring data. The signature value may be a fifth-generation message digest algorithm (MD5) signature value.

[0120] Step 302: Store the target mapping relationship between the signature value and the unitized monitoring data in the first database through the data processing node.

[0121] In an embodiment of the present application, the data processing node stores the target mapping relationship between the signature value and the unitized monitoring data in a first database. The first database is a database for storing the mapping relationship. The first database can be a relational database. For example, the first database is a MySQL database.

[0122] Step 303: The data processing node determines the target point-edge relationship of the link included in the unitized monitoring data according to the signature value corresponding to each link node.

[0123] In an embodiment of the present application, for each link, the data processing node determines the call relationship between each link node included in the link based on the signature value corresponding to each link node included in the link. Then, the data processing node determines the point-edge relationship of the link based on the call relationship between each link node included in the link. Then, the data processing node combines the point-edge relationship of each link into the target point-edge relationship of the link included in the unitized monitoring data.

[0124] Step 304: Store the target point-edge relationship in the second database through the data processing node.

[0125] In an embodiment of the present application, the data processing node stores the target vertex-edge relationship in a second database. The second database is a database for storing vertex-edge relationships. The second database may be a graph database. For example, the second database is a neo4j graph database.

[0126] In the above data processing method, the data processing node performs topological calculation on the unitized monitoring data to obtain the signature value corresponding to each link node contained in the unitized monitoring data; the data processing node stores the target mapping relationship between the signature value and the unitized monitoring data in the first database; the data processing node determines the target point-edge relationship of the link contained in the unitized monitoring data based on the signature value corresponding to each link node; and the data processing node stores the target point-edge relationship in the second database. In this way, based on the link data, the topological data is processed and stored in a classified manner. The transaction links can be stored and displayed in the form of a table and in the form of a topological map. The transaction links are clearer and more specific, so that when monitoring the unitized system, the location of the abnormal transaction link can be determined more quickly, thereby improving the efficiency of the unitized system monitoring.

[0127] In one embodiment, Figure 4 As shown, based on the preset sending strategy, the specific process of sending the unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node includes the following steps:

[0128] Step 401: sending unitized monitoring information corresponding to the node cache unitized monitoring data via a gateway.

[0129] In an embodiment of the present application, the gateway sending node uses a cache mechanism to temporarily store the unitized monitoring information corresponding to the unitized monitoring data. In one example, the gateway sending node uses a container to temporarily store the unitized monitoring information corresponding to the unitized monitoring data. For example, the container can be a hashmap.

[0130] Step 402: The unitized monitoring information is packaged through the gateway sending node to obtain the packaged unitized monitoring information.

[0131] In an embodiment of the present application, the gateway sending node classifies and organizes the unitized monitoring information, packages the unitized monitoring information of the same transaction type, and obtains the packaged unitized monitoring information.

[0132] In one example, for each transaction type, the gateway sending node collates the unitized monitoring information for the same transaction within that transaction type. The gateway sending node then packages the unitized monitoring information for different transactions within that transaction type according to the transaction attribute format, obtaining the packaged unitized monitoring information corresponding to that transaction type. The gateway sending node then combines the packaged unitized monitoring information for each transaction type into a packaged unitized monitoring information. Transaction attributes include, but are not limited to, transaction duration and transaction amount.

[0133] Step 403: The gateway sending node compresses the packaged unitized monitoring information to obtain sending data, and sends the sending data.

[0134] In an embodiment of the present application, the gateway sending node compresses the packaged unitized monitoring information to obtain transmission data. In one embodiment, the gateway sending node compresses the packaged unitized monitoring information using Java compression to obtain transmission data. The gateway sending node then transmits the transmission data. In one example, the gateway sending node transmits the transmission data to the gateway receiving node.

[0135] In the above data processing method, the gateway sending node caches the unitized monitoring information corresponding to the unitized monitoring data; the gateway sending node packages the unitized monitoring information to obtain packaged unitized monitoring information; the gateway sending node compresses the packaged unitized monitoring information to obtain transmission data, and then transmits the transmission data. In this way, the unitized monitoring information is packaged and compressed sequentially, and the compressed transmission data is transmitted, which reduces the size of the transmission content, not only reducing transmission costs but also improving transmission efficiency, thereby improving the data processing efficiency of the unitized monitoring system.

[0136] In one embodiment, Figure 5 As shown, based on the preset receiving strategy, the specific process of receiving the unitized monitoring information through the gateway receiving node includes the following steps:

[0137] Step 501: consuming and sending data via a gateway receiving node.

[0138] In an embodiment of the present application, the gateway receiving node consumes the transmission data. In one example, the gateway receiving node consumes the transmission data sent by the gateway sending node.

[0139] Step 502: Decompress the transmission data through the gateway receiving node to obtain the decompressed transmission data.

[0140] In an embodiment of the present application, the gateway receiving node decompresses the transmission data to obtain the decompressed transmission data. In one example, the gateway receiving node decompresses the transmission data using Java decompression to obtain the decompressed transmission data.

[0141] Step 503: The decompressed transmission data is unpacked by the gateway receiving node to obtain unitized monitoring information.

[0142] In an embodiment of the present application, the gateway receiving node unpacks the decompressed sending data, pieces together the data before packaging according to the same transaction, and restores the unitized monitoring information.

[0143] In the above data processing method, the gateway receiving node consumes the transmitted data; the gateway receiving node decompresses the transmitted data to obtain decompressed transmitted data; and the gateway receiving node unpacks the decompressed transmitted data to obtain unitized monitoring information. In this way, the gateway sending node sequentially packages and compresses the unitized monitoring information, and the gateway receiving node sequentially decompresses and unpacks the consumed transmitted data. This can reduce the time required to adapt the unitized monitoring information due to the unitized monitoring information coming from multiple different resources, thereby improving the data processing efficiency of the unitized monitoring system.

[0144] In one embodiment, the specific process of sending the sending data includes the following steps: sending the sending data to the message middleware.

[0145] In an embodiment of the present application, the gateway sending node sends the data to a message middleware, for example, Kafka.

[0146] The specific process of consuming and sending data through the gateway receiving node includes the following steps: consuming and sending data from the message middleware through the gateway receiving node.

[0147] In an embodiment of the present application, the gateway receiving node consumes and sends data from the message middleware in chronological order from old to new.

[0148] In the above data processing method, the sent data is sent to the message-based middleware; the sent data is consumed from the message-based middleware by the gateway receiving node. This allows the gateway sending node to send the sent data to the message-based middleware, and the gateway receiving node to consume the sent data from the message-based middleware. This buffers the sent data, preventing data loss caused by untimely data consumption by the gateway receiving node, and improving the accuracy of data processing in the unitized monitoring system. Furthermore, the use of the message-based middleware allows the same batch of sent data to be consumed simultaneously by multiple gateway receiving nodes, improving consumption capacity.

[0149] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0150] Based on the same inventive concept, the present application also provides a data processing system for implementing the aforementioned data processing method. The implementation solution provided by this system is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more data processing system embodiments provided below can be found in the above-mentioned limitations on the data processing method and will not be repeated here.

[0151] In one embodiment, a data processing system is provided. Figure 1 As shown, the data processing system may be specifically a unitized monitoring system 100, comprising: a collection terminal 102, a gateway sending node 104, a gateway receiving node 106 and a data processing node 108, wherein:

[0152] The collection terminal 102 is configured to obtain unitized monitoring data and determine the unitized indicator type of the unitized monitoring data according to a preset unitized indicator classification rule; the unitized monitoring data includes unitized metadata and monitoring data corresponding to the unitized metadata;

[0153] The gateway sending node 104 is configured to send the unitized monitoring information corresponding to the unitized monitoring data based on a preset sending strategy; the unitized monitoring information includes the unitized monitoring data and a type identifier; the type identifier is used to indicate the unitized indicator type of the unitized monitoring data;

[0154] The gateway receiving node 106 is configured to receive the unitized monitoring information based on a preset receiving strategy;

[0155] The data processing node 108 is used to determine the target storage space of the unitized monitoring data according to the type identification and preset storage rules of the unitized monitoring data, and to store and process the unitized monitoring data; the unitized monitoring data is used to monitor the unitized system.

[0156] Optionally, the data processing node 108 is specifically configured to:

[0157] Based on a preset correspondence between the unitized indicator type and the target storage space, the target storage space of the unitized monitoring data is determined, and the unitized monitoring data is stored in the target storage space.

[0158] Optionally, when the unitized indicator type of the unitized monitoring data is a link type, the data processing node 108 is further configured to:

[0159] Performing topological calculation on the unitized monitoring data to obtain a signature value corresponding to each link node contained in the unitized monitoring data;

[0160] Storing a target mapping relationship between the signature value and the unitized monitoring data in a first database;

[0161] Determining the target point-edge relationship of the link included in the unitized monitoring data according to the signature value corresponding to each link node;

[0162] The target point-edge relationship is stored in a second database.

[0163] Optionally, the gateway sending node 104 is specifically configured to:

[0164] caching unitized monitoring information corresponding to the unitized monitoring data;

[0165] Packaging the unitized monitoring information to obtain packaged unitized monitoring information;

[0166] The packaged unitized monitoring information is compressed to obtain sending data, and the sending data is sent.

[0167] Optionally, the gateway receiving node 106 is specifically configured to:

[0168] Consume the sent data;

[0169] Decompressing the transmission data to obtain decompressed transmission data;

[0170] The decompressed sending data is unpacked to obtain the unitized monitoring information.

[0171] Optionally, the gateway sending node 104 is specifically configured to:

[0172] Sending the sending data to the message middleware;

[0173] The gateway receiving node 106 is specifically configured to:

[0174] The sent data is consumed from the message middleware.

[0175] Based on the same inventive concept, the present application also provides a data processing device for implementing the aforementioned data processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more data processing device embodiments provided below can be found in the above-mentioned limitations on the data processing method and will not be repeated here.

[0176] In one embodiment, Figure 6 As shown, a data processing device 600 is provided. The device is applied to a unitized monitoring system. The unitized monitoring system includes a collection terminal, a gateway sending node, a gateway receiving node, and a data processing node. The device includes: a collection module 610, a sending module 620, a receiving module 630, and a data processing module 640, wherein:

[0177] The acquisition module 610 is configured to acquire unitized monitoring data through the acquisition terminal and determine the unitized indicator type of the unitized monitoring data according to a preset unitized indicator classification rule; the unitized monitoring data includes unitized metadata and monitoring data corresponding to the unitized metadata;

[0178] a sending module 620 configured to send, based on a preset sending strategy, unitized monitoring information corresponding to the unitized monitoring data via the gateway sending node; the unitized monitoring information comprising the unitized monitoring data and a type identifier; the type identifier being used to indicate a unitized indicator type of the unitized monitoring data;

[0179] The receiving module 630 is configured to receive the unitized monitoring information through the gateway receiving node based on a preset receiving strategy;

[0180] The data processing module 640 is used to determine the target storage space of the unitized monitoring data and perform storage processing on the unitized monitoring data according to the type identification and preset storage rules of the unitized monitoring data through the data processing node; the unitized monitoring data is used to monitor the unitized system.

[0181] Optionally, the data processing module 640 is specifically configured to:

[0182] Based on the corresponding relationship between the preset unitized indicator type and the target storage space, the target storage space of the unitized monitoring data is determined by the data processing node, and the unitized monitoring data is stored in the target storage space.

[0183] Optionally, when the unitized indicator type of the unitized monitoring data is a link type, the apparatus 600 further includes:

[0184] a calculation module, configured to perform topological calculation on the unitized monitoring data through the data processing node to obtain a signature value corresponding to each link node contained in the unitized monitoring data;

[0185] a first storage module, configured to store a target mapping relationship between the signature value and the unitized monitoring data in a first database via the data processing node;

[0186] a determination module, configured to determine, through the data processing node and based on the signature value corresponding to each link node, a target point-edge relationship of the link included in the unitized monitoring data;

[0187] The second storage module is used to store the target point-edge relationship in a second database through the data processing node.

[0188] Optionally, the sending module 620 is specifically configured to:

[0189] caching the unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node;

[0190] Packaging the unitized monitoring information through the gateway sending node to obtain packaged unitized monitoring information;

[0191] The packaged unitized monitoring information is compressed by the gateway sending node to obtain sending data, and the sending data is sent.

[0192] Optionally, the receiving module 630 is specifically configured to:

[0193] consuming the sent data via the gateway receiving node;

[0194] decompressing the transmission data through the gateway receiving node to obtain decompressed transmission data;

[0195] The decompressed sending data is unpacked by the gateway receiving node to obtain the unitized monitoring information.

[0196] Optionally, the sending module 620 is specifically configured to:

[0197] Sending the sending data to the message middleware;

[0198] The receiving module is specifically configured to:

[0199] The sending data is consumed from the message middleware through the gateway receiving node.

[0200] Each module in the above-mentioned data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0201] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0202] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0203] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0204] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0205] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0206] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0207] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0208] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0209] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data processing method, characterized in that: The method is applied to a unitized monitoring system, which includes a collection terminal, a gateway sending node, a gateway receiving node, and a data processing node; the method includes: Acquire unitized monitoring data through the acquisition end, and determine the unitized indicator type of the unitized monitoring data according to a preset unitized indicator classification rule; the unitized monitoring data includes unitized metadata and monitoring data corresponding to the unitized metadata; Based on a preset sending strategy, the unitized monitoring information corresponding to the unitized monitoring data is sent through the gateway sending node; the unitized monitoring information includes the unitized monitoring data and a type identifier; the type identifier is used to indicate the unitized indicator type of the unitized monitoring data; Based on a preset receiving strategy, receiving the unitized monitoring information through the gateway receiving node; The data processing node determines a target storage space for the unitized monitoring data according to a type identifier of the unitized monitoring data and a preset storage rule, and performs storage processing on the unitized monitoring data; the unitized monitoring data is used to monitor the unitized system; The determining, by the data processing node, a target storage space for the unitized monitoring data according to the type identifier of the unitized monitoring data and a preset storage rule, and performing storage processing on the unitized monitoring data includes: Based on a preset correspondence between the unitized indicator type and the target storage space, determining the target storage space of the unitized monitoring data through the data processing node, and storing the unitized monitoring data in the target storage space; The sending of the unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node based on the preset sending strategy includes: caching the unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node; Packaging the unitized monitoring information through the gateway sending node to obtain packaged unitized monitoring information; The packaged unitized monitoring information is compressed by the gateway sending node to obtain sending data, and the sending data is sent.

2. The method according to claim 1, characterized in that In a case where the unitized indicator type of the unitized monitoring data is a link type, the method further includes: Performing topological calculation on the unitized monitoring data by the data processing node to obtain a signature value corresponding to each link node included in the unitized monitoring data; storing, by the data processing node, a target mapping relationship between the signature value and the unitized monitoring data in a first database; Determining, by the data processing node according to the signature value corresponding to each link node, the target point edge relationship of the link included in the unitized monitoring data; The target point-edge relationship is stored in a second database through the data processing node.

3. The method according to claim 1, characterized in that The receiving of the unitized monitoring information by the gateway receiving node based on a preset receiving strategy includes: consuming the sent data via the gateway receiving node; decompressing the transmission data through the gateway receiving node to obtain decompressed transmission data; The decompressed sending data is unpacked by the gateway receiving node to obtain the unitized monitoring information.

4. The method according to claim 3, characterized in that The sending of the sending data comprises: Sending the sending data to the message middleware; The consuming of the sent data by the gateway receiving node includes: The sending data is consumed from the message middleware through the gateway receiving node.

5. A data processing system, characterized in that: The system comprises: The collection end is used to obtain unitized monitoring data and determine the unitized indicator type of the unitized monitoring data according to a preset unitized indicator classification rule; the unitized monitoring data includes unitized metadata and monitoring data corresponding to the unitized metadata; The gateway sending node is used to send the unitized monitoring information corresponding to the unitized monitoring data based on a preset sending strategy; the unitized monitoring information includes the unitized monitoring data and a type identifier; the type identifier is used to indicate the unitized indicator type of the unitized monitoring data; A gateway receiving node, configured to receive the unitized monitoring information based on a preset receiving strategy; a data processing node, configured to determine a target storage space for the unitized monitoring data according to a type identifier of the unitized monitoring data and a preset storage rule, and to perform storage processing on the unitized monitoring data; the unitized monitoring data is used to monitor the unitized system; The data processing node is specifically configured to determine the target storage space for the unitized monitoring data through the data processing node based on a preset correspondence between the unitized indicator type and the target storage space, and store the unitized monitoring data in the target storage space; The gateway sending node is specifically configured to cache the unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node; Packaging the unitized monitoring information through the gateway sending node to obtain packaged unitized monitoring information; The packaged unitized monitoring information is compressed by the gateway sending node to obtain sending data, and the sending data is sent.

6. A data processing device, characterized in that: The device is applied to a unitized monitoring system, which includes a collection terminal, a gateway sending node, a gateway receiving node, and a data processing node; the device includes: an acquisition module, configured to acquire unitized monitoring data through the acquisition terminal and determine the unitized indicator type of the unitized monitoring data according to a preset unitized indicator classification rule; the unitized monitoring data includes unitized metadata and monitoring data corresponding to the unitized metadata; A sending module, configured to send, based on a preset sending strategy, unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node; the unitized monitoring information includes unitized monitoring data and a type identifier; the type identifier is used to indicate a unitized indicator type of the unitized monitoring data; A receiving module, configured to receive the unitized monitoring information through the gateway receiving node based on a preset receiving strategy; a data processing module, configured to determine, through the data processing node, a target storage space for the unitized monitoring data according to a type identifier of the unitized monitoring data and a preset storage rule, and to perform storage processing on the unitized monitoring data; the unitized monitoring data is used to monitor the unitized system; The data processing module is specifically configured to determine the target storage space of the unitized monitoring data through the data processing node based on a preset correspondence between the unitized indicator type and the target storage space, and store the unitized monitoring data in the target storage space; The sending module is specifically configured to cache the unitized monitoring information corresponding to the unitized monitoring data through the gateway sending node; Packaging the unitized monitoring information through the gateway sending node to obtain packaged unitized monitoring information; The packaged unitized monitoring information is compressed by the gateway sending node to obtain sending data, and the sending data is sent.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images