Data processing method, device and equipment
By receiving and classified log data in a distributed system, and depositing it into the database, the problem of low query efficiency in the traditional distributed system architecture is solved, and more efficient log data processing is achieved.
Patent Information
- Application Number
- CN202411411184.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-01-24
AI Technical Summary
The traditional distributed system architecture is less efficient when processing log queries, and it is necessary to query the target log information from each node in turn.
By receiving the log data set of the distributed system, the log data is classified according to the preset classification strategy, forming multiple types of log data sets, and sending the same type of log data set to Enter the database for subsequent query and processing according to the log type.
It improved the query efficiency and processing efficiency of log data. Through centralized management and category storage, the log query and processing process was simplified.
Smart Images
Figure CN119357148A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data processing technology, and in particular to a data processing method, device and equipment. Background Art
[0002] In order to meet the massive information query needs and improve the fault tolerance and availability of information systems, most companies currently use distributed system architecture to process information. Under the traditional distributed system architecture, when there is a log query demand, it is necessary to query the target log information from each node of the distributed system in turn and perform corresponding analysis and processing operations on the target log information. This processing method is inefficient. Summary of the invention
[0003] The present application provides a data processing method, apparatus and device for improving the processing efficiency of log data in a distributed system.
[0004] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a data processing device, wherein the data processing device is connected to a distributed system, and each node in the distributed system stores customer information of a financial institution; the method comprises: receiving a log data set from the distributed system, wherein the log data set comprises log data generated by each node; classifying the log data set according to a preset classification strategy to obtain multiple types of log data subsets, wherein the classification strategy is used to indicate classification according to the business service or operation corresponding to each log data; sending the log data subsets of the same type to the target storage area corresponding to the type in the message queue, and storing the log data subsets in the target storage area in the database; obtaining a data processing request, and according to the log type indicated by the data processing request, obtaining target log data that conforms to the log type from the database, and processing the target log data.
[0005] In an embodiment of the present application, the log data generated by each node in the distributed system is aggregated into a log data set, which is equivalent to centrally managing the log data generated by all nodes in the distributed system, which is convenient for subsequent log queries. In addition, before storing the log data set, the log data set is classified and stored according to the business service or operation corresponding to each log data, and the log data of the same type is stored in the corresponding storage area. Then, when performing log data processing operations later, the target log data of the corresponding log type can be obtained from the database according to the log type indicated by the data processing request, which is conducive to improving the efficiency of log data query and log data processing.
[0006] In a possible implementation, sending the same type of log data subsets to the target storage area corresponding to the type in the message queue includes: partitioning the target storage area according to a preset partitioning strategy to obtain multiple sub-areas; the partitioning strategy is used to indicate partitioning according to the number of unique identifiers of log data in the same type of log data subset; associating the unique identifier of each log data in the log data subset with a sub-area to obtain a corresponding relationship between the log data subset and the multiple sub-areas; based on the corresponding relationship between the unique identifier of each log data in the log data subset and the multiple sub-areas and the unique identifier of each log data in the log data subset, storing each log data in the log data subset into a corresponding sub-area.
[0007] In one possible implementation, the multiple types of log data include at least a first type, a second type, and a third type, wherein the first type is used to indicate transaction-type log data, the second type is used to indicate customer information maintenance-type log data, and the third type is used to indicate error-type log data.
[0008] In one possible implementation, the data processing request is used to indicate whether there is an abnormality in the distributed system within a preset time period; according to the log type indicated by the data processing request, target log data that conforms to the log type is obtained from the database, and the target log data is processed, including: according to the log type, obtaining a third type of log data within the preset time period from the database as the target log data; determining whether the number of the target log data reaches a preset number; if the number reaches the preset number, generating an alarm message, and the alarm message is used to indicate that there is an abnormality in the distributed system.
[0009] In one possible implementation, each node in the distributed system is configured with a log generation tool, and the log generation tool is used to generate log data in a preset format based on the business processing operations of each node; the preset format includes a first format and a second format, the first format is used to generate log data for external business services in the distributed system, and the second format is used to generate log data for internal business services in the distributed system.
[0010] In a possible implementation, the internal business service is a batch online service; the method further includes: real-time monitoring of a first response delay for the external business service and a second response delay for the internal business service in the distributed system; if it is determined that the first response delay exceeds a first preset threshold and the second response delay does not exceed a second preset threshold, adjusting the file size of a batch file for each business service in the batch online service.
[0011] In a second aspect, an embodiment of the present application provides a data processing device, which is connected to a distributed system, and each node in the distributed system stores customer information of a financial institution; wherein the data processing device includes: a transceiver module, which is used to receive a log data set from the distributed system, and the log data set includes log data generated by each node; a classification module, which is used to classify the log data set according to a preset classification strategy to obtain multiple types of log data subsets, and the classification strategy is used to indicate classification according to the business service or operation corresponding to each log data; a storage module, which is used to send the log data subsets of the same type to the target storage area corresponding to the type in the message queue, and store the log data subsets in the target storage area into a database; the data processing module is also used to obtain a data processing request, and according to the log type indicated by the data processing request, obtain target log data that conforms to the log type from the database, and process the target log data.
[0012] In a possible implementation, the storage module is specifically used to: partition the target storage area according to a preset partitioning strategy to obtain multiple sub-areas; the partitioning strategy is used to indicate partitioning according to the number of unique identifiers of log data in the log data subset of the same type; associate the unique identifier of each log data in the log data subset with a sub-area to obtain a corresponding relationship between the log data subset and the multiple sub-areas; based on the corresponding relationship between the unique identifier of each log data in the log data subset and the multiple sub-areas and the unique identifier of each log data in the log data subset, store each log data in the log data subset in the corresponding sub-area.
[0013] In one possible implementation, the multiple types of log data include at least a first type, a second type, and a third type, wherein the first type is used to indicate transaction-type log data, the second type is used to indicate customer information maintenance-type log data, and the third type is used to indicate error-type log data.
[0014] In one possible implementation, the data processing request is used to indicate whether there is an abnormality in the distributed system within a preset time period; the data processing module is specifically used to: obtain, according to the log type, a third type of log data within the preset time period from the database as the target log data; determine whether the number of the target log data reaches a preset number; if the number reaches the preset number, generate an alarm message, and the alarm message is used to indicate that there is an abnormality in the distributed system.
[0015] In one possible implementation, each node in the distributed system is configured with a log generation tool, and the log generation tool is used to generate log data in a preset format based on the business processing operations of each node; the preset format includes a first format and a second format, the first format is used to generate log data for external business services in the distributed system, and the second format is used to generate log data for internal business services in the distributed system.
[0016] In a possible implementation, the internal business service is a batch online service; the data processing device also includes an adjustment module, which is used to: monitor in real time a first response delay for the external business service and a second response delay for the internal business service in the distributed system; if it is determined that the first response delay exceeds a first preset threshold and the second response delay does not exceed a second preset threshold, adjust the file size of the batch file of each business service in the batch online service.
[0017] In a third aspect, an embodiment of the present application provides a data processing device, comprising: at least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the method described in the first aspect and any possible implementation manner by executing the instructions stored in the memory.
[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes the method described in the first aspect and any possible implementation manner.
[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed on a computer, enables the method described in the first aspect and any possible implementation method to be implemented.
[0020] The beneficial effects of the second to fifth aspects can be referred to the contents described in the first aspect above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 A schematic diagram of an application scenario of a data processing method provided in an embodiment of the present application;
[0022] Figure 2 A system architecture diagram of a data processing method provided in an embodiment of the present application;
[0023] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;
[0024] Figure 4 A schematic diagram of a storage area of a message queue provided in an embodiment of the present application;
[0025] Figure 5 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0026] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.
[0028] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.
[0029] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.
[0030] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.
[0031] In order to better understand the technical solution provided by the embodiments of the present application, Figure 1 The application scenarios to which the embodiments of the present application are applicable are briefly described.
[0032] Please refer to Figure 1 , is a schematic diagram of an application scenario of a data processing method provided in an embodiment of the present application. Figure 1 As shown, the schematic diagram includes a distributed system 110 and a data processing device 120. The distributed system 110 includes a first node 111, a second node 112, and a third node 113. Each node in the distributed system 110 can communicate with the data processing device 120 via wired or wireless communication.
[0033] The distributed system 110 refers to a distributed system with data processing capabilities, and specifically refers to a distributed system that supports information query, information processing, and information storage capabilities. The distributed system 110 may be composed of multiple distributed computers, which are interconnected through a network, and one computer may represent a node. In the embodiment of the present application, each node in the distributed system 110 stores customer information of a financial institution, and the customer information may include the customer's contact information, ID number, customer number, etc. Figure 1 In the figure, the distributed system 110 includes three nodes for illustration, but in actual application, the number of nodes included in the distributed system 110 may be more or less, and the embodiment of the present application does not limit this. The data processing device 120 refers to a terminal device or server with data processing capability, and the terminal device includes but is not limited to a mobile phone, a personal computer (PC), a tablet computer, a laptop computer, a PDA, and a mobile internet device (MID).
[0034] In order to more clearly illustrate the interaction process between the distributed system and the data processing device, Figure 2 The system architecture diagram of a data processing method shown illustrates the interaction process between the distributed system and the data processing device.
[0035] like Figure 2 As shown, the system architecture diagram includes a distributed system 210 and a data processing device 220, and the distributed system 210 includes a first node 211, a second node 212 and a third node 213. Figure 2 The distributed system 210 shown is, for example, Figure 1 The distributed system 210 shown, Figure 2 The first node 211 shown is, for example, Figure 1 The first node 111 shown, Figure 2 The second node 212 shown is, for example, Figure 1 The second node 112 shown, Figure 3 The third node 213 shown is, for example, Figure 1 The third node 213 shown, Figure 2 The data processing device 220 shown is, for example, Figure 1 The data processing device 120 is shown.
[0036] In a possible implementation, the first node 211, the second node 212, and the third node 213 are respectively configured with a log generation tool 214. The log generation tool 214 is used to generate log data in a preset format according to the business processing operation of each node in the distributed system 210. Exemplarily, the log generation tool 214 is, for example, Spring-Log4j2, and the business processing operation is, for example, a customer information query operation, a transaction call operation, a customer information maintenance operation, etc. For example, when a business person executes a transaction, the customer information of the transaction party needs to be called. At this time, the log generation tool 214 monitors the business processing operation and generates transaction log data.
[0037] Among them, the preset format includes a first format and a second format, the first format is used to generate log data of external business services in a distributed system, and the second format is used to generate log data of internal business services in a distributed system. Generally speaking, external business services can refer to services related to customer information in a distributed system that are called by peripheral components, such as deposit and withdrawal services. Since external business services are implemented through application routing, their request method belongs to Hypertext Transfer Protocol (HTTP) requests, so the format of the request message of the external business service is http format, and accordingly, the log data generated by the log generation tool is in http format. Internal business services can refer to business services between distributed systems, such as batch online services. In this example, when a node in a distributed system receives a batch file, it is necessary to split the batch file into transactions one by one, and call online transactions according to each transaction. At this time, the log data generated by the log generation tool is in JSON format.
[0038] In a possible implementation, when the internal business service is a batch online service, the data processing device 220 may also monitor in real time the first response delay for the external business service and the second response delay for the internal business service in the distributed system. If it is determined that the first response delay exceeds the first preset threshold and the second response delay does not exceed the second preset threshold, the file size of the batch file of each business service in the batch online service is adjusted. Among them, the data processing device can adjust the file size of the batch file by sending instructions to the distributed system. That is to say, when the data processing device detects that most of the computing resources in the distributed system are called to process internal business services, resulting in insufficient computing resources for external business services and unable to meet the normal use of external business services, the concurrency of the batch files can be adjusted so that the distributed system can meet the external response and the internal batch file response at the same time.
[0039] In a possible implementation, the log generation tool 214 is also configured with log levels, which may include INFO, DEBUG, and ERROR. In the log generation tool, different log levels may be used for different log data according to business requirements. For example, customer information creation, customer information query, transaction request message, transaction response message, etc. use the INFO level. Log data generated by transaction anomalies, system errors, etc. use the ERROR level. Transaction anomalies, for example, customer maintenance transaction failures, system errors, for example, database query failures, transaction errors, etc.
[0040] In a possible implementation, the log generation tool 214 is also configured with a log structure, which may include key fields such as customer number, transaction serial number (including transaction initiator, timestamp), operation time (time of transaction request), operation type (type of transaction request), and operation result (business error, system error in transaction response result). Among them, a business error is, for example, when querying customer information, it is displayed that the customer does not exist; a system error is, for example, when querying customer information, the connection between the application and the database is disconnected. This structured log data can facilitate subsequent log parsing and query, which is conducive to improving the processing efficiency of log data.
[0041] In a possible implementation, the log generation tool 214 may be pre-configured with asynchronous logging, so that when the log generation tool 214 generates log data in each node, asynchronous log generation may be implemented. Exemplarily, high-performance asynchronous log writing may be implemented through the Disruptor, reducing synchronous blocking of the application and improving overall system performance.
[0042] The data processing device 220 includes a message queue 221, a database 222, and a data query module 223. The message queue 221 is, for example, a kafka message queue, the database 222 is, for example, an Elasticsearch database, and the data query module 223 is, for example, Kibana. In this example, the functions of data collection, processing, and forwarding between the message queue 221, the database 222, and the data query module 223 can be implemented by the Logstash tool.
[0043] Exemplarily, the data processing device 220 can receive a log data set from the distributed system 210, and classify the log data set according to a preset classification strategy, and then store the classified log data subset in a storage area matching its type in the message queue 221, and store the log data subset in the storage area in the database 222. When the data query module 223 receives a data processing request, it can obtain target log data that meets the corresponding log type from the database 222 according to the log type indicated by the data processing request, and process the target log data. The specific processing process of the log data by the data processing device 220 will be described in detail below.
[0044] Please refer to Figure 3 , is a flow chart of a data processing method provided in an embodiment of the present application. The following is a flow chart of a data processing method performed by a data processing device. Figure 3 The data processing device is, for example, Figure 1 The data processing device 120 shown, Figure 3 The distributed system involved is, for example, Figure 1 The distributed system 110 shown, Figure 3 The nodes involved are, for example, Figure 1 The first node 111, the second node 112 or the third node 113 shown. Or, Figure 3 The data processing device involved is, for example, Figure 2 The data processing device 220 shown, Figure 3 The distributed system involved is, for example, Figure 2 The distributed system 210 shown, Figure 3 The nodes involved are, for example, Figure 2 The first node 211, the second node 212 or the third node 213 shown, Figure 3 The message queue involved is, for example, Figure 2 The message queue 221 shown, Figure 3 The database involved is, for example, Figure 2 Database 222 is shown.
[0045] S301, receiving a log data set from a distributed system, where the log data set includes log data generated by each node.
[0046] Among them, each node in the distributed system stores customer information of the financial institution. Each node in the distributed system is pre-configured with a log generation tool, and the log generation tool can generate log data in each node. The specific way in which the log generation tool generates log data can be referred to the content described above, and will not be repeated here.
[0047] The data processing device receives log data sent from all nodes in the distributed system to obtain a log data set.
[0048] S302, classifying the log data set according to a preset classification strategy to obtain multiple types of log data subsets, where the classification strategy is used to indicate classification according to the business service or operation corresponding to each log data.
[0049] Among them, business services or operations include, for example, transaction call services and customer information maintenance operations, and customer information maintenance operations include, for example, customer information creation, customer information modification, customer information query, and other operations. The log data corresponding to the transaction call service is the transaction request message log data and the transaction response message log data. The log data generated by the customer information maintenance operation is the customer information maintenance log data. In particular, when an error occurs in executing the transaction call service or the customer information maintenance operation, error log data will be generated.
[0050] Based on this, in an embodiment of the present application, multiple types of log data may include at least a first type, a second type and a third type, wherein the first type is used to indicate transaction type log data, the second type is used to indicate customer information maintenance type log data, and the third type is used to indicate error type log data.
[0051] Specifically, the data processing device may classify the business services or operations corresponding to each log data to obtain multiple types of log data subsets. Exemplarily, the log data includes operation results and operation types, and the data processing device may determine the corresponding business service or operation according to the operation result or operation type of each log data. For example, the operation result corresponding to the log data of the transaction business service is a successful transaction, and the operation result corresponding to the log data of the customer information maintenance operation is a successful customer information maintenance. In particular, since the log level of the error-type log data is different from the log levels of the first type and the second type, the data processing device may classify all log data with an error level in the log data set as a third type of log data subset.
[0052] S303, sending the log data subset of the same type to the target storage area corresponding to the type in the message queue, and storing the log data subset in the target storage area in the database.
[0053] The target storage area is obtained by the data processing device according to the log type, and the log type can be the first type, the second type and the third type mentioned above. That is, the data processing device divides the storage area in the message queue into the same number of storage areas according to the number of the aforementioned multiple types of log data subsets, so that one storage area corresponds to one type.
[0054] Then, the data processing device may determine the storage area corresponding to each type in the message queue, and send the log data subset of the same type to the target storage area corresponding to the type in the message queue.
[0055] In a possible implementation, the data processing device may also partition the target storage area according to a preset partitioning strategy to obtain a plurality of sub-areas. The partitioning strategy is used to indicate partitioning according to the number of unique identifiers of the log data in the same type of log data subset, and the unique identifier is, for example, a customer number or a transaction serial number in the log data. In other words, the data processing device may store the log data of the same customer or the same transaction in a sub-area to facilitate subsequent log storage and query. After partitioning to obtain a plurality of sub-areas, the data processing device associates the unique identifier of each log data in the log data subset with a sub-area to obtain a corresponding relationship between the log data subset and the plurality of sub-areas, and then stores each log data in the log data subset in the corresponding sub-area according to the unique identifier of each log data in the log data subset.
[0056] For example, the following will be combined with Figure 4A schematic diagram of a storage area of a message queue is shown, taking multiple types including a first type, a second type and a third type, a log data subset of the first type including log data 1, log data 2, log data 3, log data 4 and log data 5, and the customer numbers of log data 1 and log data 3 are the same, and the customer numbers of log data 2 and log data 4 are the same as an example, to illustrate the storage method of the message queue in an embodiment of the present application.
[0057] like Figure 4 As shown, the storage area of the message queue is divided into a first storage area, a second storage area and a third storage area according to the first type, the second type and the third type. The data processing device determines that the number of unique identifiers (customer numbers in this example) in the first type of log data subset is 3, so the first storage area is divided into sub-area 1, sub-area 2 and sub-area 3. Sub-area 1 is associated with the customer numbers of log data 1 and 3, so log data 1 and 3 are stored in sub-area 1. Sub-area 2 is associated with the customer numbers of log data 2 and 4, so log data 2 and 4 are stored in sub-area 2. Sub-area 3 is associated with the customer number of log data 5, so log data 5 is stored in sub-area 3.
[0058] After the data processing device sends the log data in the log data collection to the message queue, the log data can be consumed from the message queue to be stored in the database. When the log data in the message queue is stored in the database, the concurrent consumption thread pool of the consumer can be configured so that multiple threads concurrently consume the log data in the message queue, and one thread can correspond to a sub-area in the message queue, thereby improving the processing efficiency of the log data.
[0059] For example, taking the message queue as Kafak and the database as Elasticsearch as an example, the data processing device can use the Logstash tool to process and forward data. Specifically, Kafka can be set as the input plug-in in the Logstash configuration file in advance, and log data can be consumed from the Kafka storage area. Configure the Logstash filter to pre-process the log data, such as parsing and structuring. Configure Elasticsearch as the output plug-in to transfer the processed log data to Elasticsearch for indexing and storage.
[0060] In this example, the data processing device creates different index templates according to different types of log data, and defines the field type and index method. Furthermore, since a large amount of log data is generated every day in the distributed system, a rolling index strategy can be adopted to create a new index every day to avoid a single index that is too large and causes query performance to deteriorate. Among them, the index can be a transaction serial number, customer number, etc., and the index of the log data involving the system alarm can be the device number or node identifier of the alarm, etc., which can be set according to actual needs.
[0061] In addition, you can also use the pre-aggregation function of Elasticsearch to pre-calculate common queries and cache the results in high-performance cache systems such as Redis to improve query response speed. For high-frequency query fields (such as customer numbers and transaction serial numbers), set up inverted indexes and word breakers to improve search efficiency.
[0062] In a possible implementation, in order to ensure that there is enough storage space in the message queue for storing log data, the consumed data in the message queue may be cleared at intervals of a preset time. The preset time may be set according to actual needs, such as 24 hours, 48 hours, etc., and this embodiment of the application does not limit this.
[0063] In one possible implementation, in order to make the message queue more suitable for the application scenario described in the embodiment of the present application, the relevant parameters of the producer and consumer may be adjusted and optimized. Specifically, the reliability-related parameters in the message queue may be adjusted to improve the reliability of the message queue in processing log data and avoid the loss of important log data or data inconsistency.
[0064] S304, obtaining a data processing request, and according to the log type indicated by the data processing request, obtaining target log data that matches the log type from a database, and processing the target log data.
[0065] The data processing request may be used to instruct to query the log data generated by the target transaction or target customer within a preset time period, or the data processing request may also be used to instruct to determine whether there is an abnormality in the distributed system within a preset time period. The data processing request may be input by a user or received from other devices, and the embodiments of the present application do not limit this.
[0066] When a data processing request is used to indicate a query for log data generated by a target transaction or a target customer within a preset time period, the log data generated within the preset time period can be obtained from the database based on the transaction serial number of the target transaction or the customer number of the target customer, and the queried log data can be displayed to the user.
[0067] When the data processing request is used to indicate whether the distributed system has an abnormality within a preset time period, the data processing device can obtain the third type of log data within the preset time period from the database according to the log type indicated by the data processing request as the target log data, wherein the third type of log data is error-reporting log data. The data processing device determines whether the number of target log data reaches the preset number. If the number of target log data reaches the preset number, an alarm message is generated and sent to relevant operation and maintenance personnel so that the operation and maintenance personnel can handle the abnormal situation in the distributed system in a timely manner based on the alarm message, wherein the alarm message is used to indicate that the distributed system has an abnormality. The preset number can be set according to actual needs. For example, for system-type error logs, the preset number can be 0, which is not limited in the embodiments of the present application. If the number of target log data does not reach the preset number, no alarm processing is performed.
[0068] Exemplarily, the data processing device can implement a data query function through Kibana, and the user can input query information on the visual interface of Kibana, and then Kibana obtains log data matching the query information from the database based on the query information input by the user.
[0069] In this example, you can also configure the index name of the log data to be monitored (usually the same as the index name in the database) or log type in Kibana according to monitoring requirements, and configure the time range to be monitored, so that Kibana can display log data based on time filtering to determine whether to trigger an alarm.
[0070] Based on the same inventive concept, an embodiment of the present application provides a data processing device, which is used to implement any of the above data processing methods, for example Figure 3 The data processing method shown, and the device can also realize the functions of the data processing device mentioned above.
[0071] Please refer to Figure 5 , is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 5 As shown, the data processing device 500 includes a transceiver module 501, a classification module 502, a storage module 503, and a data processing module 504. The data processing device 500 can be connected to a distributed system, and each node in the distributed system stores customer information of a financial institution.
[0072] Exemplarily, the transceiver module 501 is used to receive a log data set from a distributed system, and the log data set includes log data generated by each node; the classification module 502 is used to classify the log data set according to a preset classification strategy to obtain multiple types of log data subsets, and the classification strategy is used to indicate classification according to the business service or operation corresponding to each log data; the storage module 503 is used to send the log data subsets of the same type to the target storage area corresponding to the type in the message queue, and store the log data subsets in the target storage area into the database; the data processing module 504 is also used to obtain a data processing request, and according to the log type indicated by the data processing request, obtain the target log data that meets the log type from the database, and process the target log data.
[0073] In a possible implementation, the storage module 503 is specifically used to: partition the target storage area according to a preset partitioning strategy to obtain multiple sub-areas; the partitioning strategy is used to indicate partitioning according to the number of unique identifiers of log data in the same type of log data subset; associate the unique identifier of each log data in the log data subset with a sub-area to obtain a corresponding relationship between the log data subset and the multiple sub-areas; based on the corresponding relationship between the unique identifier of each log data in the log data subset and the multiple sub-areas and the unique identifier of each log data in the log data subset, store each log data in the log data subset in the corresponding sub-area.
[0074] In one possible implementation, multiple types of log data include at least a first type, a second type, and a third type, wherein the first type is used to indicate transaction-type log data, the second type is used to indicate customer information maintenance-type log data, and the third type is used to indicate error-type log data.
[0075] In one possible implementation, a data processing request is used to indicate whether an abnormality exists in a distributed system within a preset time period; a data processing module 504 is specifically used to: obtain a third type of log data within a preset time period from a database as target log data according to the log type; determine whether the number of target log data reaches a preset number; if the number reaches the preset number, generate an alarm message, and the alarm message is used to indicate that an abnormality exists in the distributed system.
[0076] In one possible implementation, each node in the distributed system is configured with a log generation tool, which is used to generate log data in a preset format based on the business processing operations of each node; the preset format includes a first format and a second format, the first format is used to generate log data for external business services in the distributed system, and the second format is used to generate log data for internal business services in the distributed system.
[0077] In a possible implementation, the internal business service is a batch online service; the data processing device 500 further includes an adjustment module 505, the adjustment module 505 being used to: monitor in real time the first response delay for the external business service and the second response delay for the internal business service in the distributed system; if it is determined that the first response delay exceeds a first preset threshold and the second response delay does not exceed a second preset threshold, adjust the file size of the batch file of each business service in the batch online service.
[0078] Based on the same inventive concept, an embodiment of the present application provides a data processing device, which is used to implement any of the above data processing methods, for example Figure 3 The data processing method shown, and the device can also realize the functions of the data processing device mentioned above.
[0079] Please refer to Figure 6 , is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 6 As shown, the data processing device 600 includes at least one processor 601 and a memory 602 communicatively connected to the at least one processor 601 .
[0080] The processor 601 may be a general-purpose processor or a dedicated processor. The processor 601 may include, for example, a baseband processor or a central processing unit. The baseband processor may be used to process the communication protocol and the communication data. The central processing unit may be used to control the data processing device 600, execute the software program and / or process the data. Different processors may be independent devices or may be arranged in one or more processing circuits, for example, integrated in one or more dedicated integrated circuits.
[0081] In one embodiment, the memory 602 stores instructions that can be executed by at least one processor 601. The at least one processor 601 implements the functions of the data processing device as described above by executing the instructions stored in the memory 602, and accordingly, can also implement the steps performed by the data processing device as described above.
[0082] In this embodiment, the data processing device 600 can also implement the functions of the previous text data processing device 500, and at least one processor 601 in the data processing device 600 can also implement the functions of the previous text transceiver module 501, the classification module 502, the storage module 503, the data processing module 504 and the adjustment module 505.
[0083] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes any of the above-mentioned data processing methods, for example, Figure 3 The data processing method shown.
[0084] Based on the same inventive concept, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed on a computer, enables any of the above data processing methods, for example, Figure 3 The data processing method shown is implemented.
[0085] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0086] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0087] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0089] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A data processing method, characterized in that: The data processing method is applied to a data processing device, the data processing device is connected to a distributed system, and each node in the distributed system stores customer information of a financial institution; the method comprises: Receiving a log data set from the distributed system, the log data set including log data generated by each node; Classifying the log data set according to a preset classification strategy to obtain multiple types of log data subsets, wherein the classification strategy is used to indicate classification according to the business service or operation corresponding to each log data; Sending a subset of log data of the same type to a target storage area corresponding to the type in the message queue, and storing the subset of log data in the target storage area in the database; A data processing request is obtained, and according to the log type indicated by the data processing request, target log data that matches the log type is obtained from the database, and the target log data is processed.
2. The method according to claim 1, characterized in that The sending of the same type of log data subsets to the target storage area corresponding to the type in the message queue includes: According to a preset partitioning strategy, the target storage area is partitioned to obtain a plurality of sub-areas; the partitioning strategy is used to indicate that the partitioning is performed according to the number of unique identifiers of the log data in the subset of log data of the same type; Associating a unique identifier of each log data in the log data subset with a sub-region to obtain a corresponding relationship between the log data subset and the multiple sub-regions; According to the correspondence between the unique identifier of each log data in the log data subset and the multiple sub-areas and the unique identifier of each log data in the log data subset, each log data in the log data subset is stored in a corresponding sub-area.
3. The method according to claim 2, characterized in that The multiple types of log data include at least a first type, a second type and a third type, wherein the first type is used to indicate transaction type log data, the second type is used to indicate customer information maintenance type log data, and the third type is used to indicate error type log data.
4. The method according to any one of claims 1 to 3, characterized in that: The data processing request is used to indicate whether there is an abnormality in the distributed system within a preset time period; The acquiring, according to the log type indicated by the data processing request, target log data that matches the log type from the database, and processing the target log data, includes: According to the log type, obtaining log data of a third type within the preset time period from the database as the target log data; Determining whether the amount of the target log data reaches a preset amount; If the number reaches the preset number, an alarm message is generated, and the alarm message is used to indicate that an abnormality exists in the distributed system.
5. The method according to any one of claims 1 to 3, characterized in that: Each node in the distributed system is configured with a log generation tool, and the log generation tool is used to generate log data in a preset format according to the business processing operation of each node; The preset format includes a first format and a second format, the first format is used to generate log data of external business services in the distributed system, and the second format is used to generate log data of internal business services in the distributed system.
6. The method according to claim 5, characterized in that The internal business service is a batch online service; the method further comprises: Monitoring in real time a first response delay for the external business service and a second response delay for the internal business service in the distributed system; If it is determined that the first response delay exceeds a first preset threshold and the second response delay does not exceed a second preset threshold, the file size of the batch file of each business service in the batch online service is adjusted.
7. A data processing device, characterized in that: The data processing device is connected to a distributed system, and each node in the distributed system stores customer information of a financial institution; wherein the data processing device comprises: A transceiver module, configured to receive a log data set from the distributed system, wherein the log data set includes log data generated by each node; A classification module, used to classify the log data set according to a preset classification strategy to obtain multiple types of log data subsets, wherein the classification strategy is used to indicate classification according to the business service or operation corresponding to each log data; A storage module, used for sending a subset of log data of the same type to a target storage area corresponding to the type in the message queue, and storing the subset of log data in the target storage area in the database; The data processing module is also used to obtain a data processing request, and according to the log type indicated by the data processing request, obtain target log data that conforms to the log type from the database, and process the target log data.
8. The device according to claim 7, characterized in that The storage module is specifically used for: According to a preset partitioning strategy, the target storage area is partitioned to obtain a plurality of sub-areas; the partitioning strategy is used to indicate that the partitioning is performed according to the number of unique identifiers of the log data in the subset of log data of the same type; Associating a unique identifier of each log data in the log data subset with a sub-region to obtain a corresponding relationship between the log data subset and the multiple sub-regions; According to the correspondence between the unique identifier of each log data in the log data subset and the multiple sub-areas and the unique identifier of each log data in the log data subset, each log data in the log data subset is stored in a corresponding sub-area.
9. The device according to claim 8, characterized in that The multiple types of log data include at least a first type, a second type and a third type, wherein the first type is used to indicate transaction type log data, the second type is used to indicate customer information maintenance type log data, and the third type is used to indicate error type log data.
10. The device according to any one of claims 7 to 9, characterized in that: The data processing request is used to indicate whether there is an abnormality in the distributed system within a preset time period; The data processing module is specifically used for: According to the log type, obtaining log data of a third type within the preset time period from the database as the target log data; Determining whether the amount of the target log data reaches a preset amount; If the number reaches the preset number, an alarm message is generated, and the alarm message is used to indicate that an abnormality exists in the distributed system.
11. The device according to any one of claims 7 to 9, characterized in that: Each node in the distributed system is configured with a log generation tool, and the log generation tool is used to generate log data in a preset format according to the business processing operation of each node; The preset format includes a first format and a second format, the first format is used to generate log data of external business services in the distributed system, and the second format is used to generate log data of internal business services in the distributed system.
12. The device according to claim 11, characterized in that The internal business service is a batch online adjustment service; the data processing device also includes an adjustment module, and the adjustment module is used to: Monitoring in real time a first response delay for the external business service and a second response delay for the internal business service in the distributed system; If it is determined that the first response delay exceeds a first preset threshold and the second response delay does not exceed a second preset threshold, the file size of the batch file of each business service in the batch online service is adjusted.
13. A data processing device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method according to any one of claims 1 to 6 by executing the instructions stored in the memory.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.
15. A computer program product, characterized in that The method comprises computer instructions, which, when executed on a computer, enable the method according to any one of claims 1 to 6 to be implemented.