Log data processing method, device, computer equipment, and system

Abstract

The present application proposes a log data processing method, device, computer equipment, system, computer-readable storage medium and computer program product, which relate to the field of cloud computing technology. The log data processing method proposed in the present application is applied to a log processing cloud server in a log data processing system, and the log data processing system also includes an application server that runs an application service. After the application server runs the application service and generates raw data, the application server sends the collected raw data to the log processing cloud server. After the log processing cloud server obtains the raw data sent by the target application server, it queries the target log data processing policy corresponding to the target application service according to the correspondence between the pre-stored configured application service and the log data processing policy, and converts the raw data into log data according to the target log data processing policy.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to log data processing methods, devices, computer equipment, and systems. Background Art

[0002] Log data plays a crucial role in internet applications. Troubleshooting, data analysis, and intranet security monitoring all rely on log data. The requirements, format, and content of log data vary across different application scenarios. Generally, the collected raw data must be processed according to the requirements to generate the corresponding log data.

[0003] In related technologies, when an application service is deployed on a server, a log data processing service is also deployed. When the application server runs the application service and generates raw data, the log data processing service on the application server converts the raw data into corresponding log data according to the configuration, and then stores it in the corresponding database. Raw data refers to the unprocessed data generated when the application server runs the application service, which corresponds to the expected form of log data. Raw data refers to data in an unexpected form.

[0004] However, the resources of the application server are limited, and the log data processing service occupies part of the resources of the application server, which limits the resources of the application server and reduces the service quality. Summary of the Invention

[0005] Based on this, it is necessary to provide a log data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can ensure application service resources and service quality in response to the above technical problems.

[0006] In a first aspect, the present application provides a log data processing method. The method is applied to a log processing cloud server in a log data processing system, wherein the log data processing system also includes an application server running an application service; the method comprises:

[0007] Obtain the original data generated by the target application server running the target application service;

[0008] According to the pre-stored correspondence between application services and log data processing strategies, query the target log data processing strategy corresponding to the target application service;

[0009] The original data is converted into log data according to the target log data processing strategy.

[0010] In one embodiment, obtaining the original data generated by the target application server running the target application service includes:

[0011] Receiving compressed data sent by a target application server; the compressed data is obtained by the target application server compressing original data generated by running the target application service;

[0012] The compressed data is decompressed to obtain original data generated by the target server running the target application service.

[0013] In one embodiment, converting the original data into log data according to the log data processing strategy includes:

[0014] Determining a target conversion type corresponding to the original data according to the log data processing strategy;

[0015] Determine, from pre-created process groups, a plurality of process groups corresponding to the target conversion type, and select, from the plurality of process groups, a target process group whose working state is a load-unsaturated state;

[0016] The target process group is called to convert the original data into log data.

[0017] In one embodiment, the method further comprises:

[0018] Query the working status of multiple process groups corresponding to the target conversion type;

[0019] When the working states of multiple process groups corresponding to the target conversion type are all in a load oversaturation state, a process group with a conversion type of the target conversion type is created.

[0020] In one embodiment, the method further comprises:

[0021] When the working states of multiple process groups corresponding to the target conversion type are all in an overloaded state, the application service corresponding to the target conversion type is determined, and a current limiting instruction is sent to the server where the application service is located. The current limiting instruction is used to instruct the server where the application service is located to reduce the rate of sending original data.

[0022] In one embodiment, the method further comprises:

[0023] When the number of process groups in a load-unsaturated state among the multiple process groups corresponding to the target conversion type reaches a preset threshold, at least one process group in a load-unsaturated state among the multiple process groups corresponding to the target conversion type is deleted.

[0024] In a second aspect, the present application also provides a log data processing system. The log data processing system includes at least an application server running an application service and a log processing cloud server; the method includes:

[0025] The application server is used to collect raw data generated by running application services and send the raw data to the log processing cloud server;

[0026] The log processing cloud server is used to obtain the original data generated by the application server running the application service; query the target log data processing policy corresponding to the target application service based on the correspondence between the pre-stored configured application service and the log data processing policy; and convert the original data into log data according to the target log data processing policy.

[0027] In a third aspect, the present application further provides a log data processing device. The device is applied to a log processing cloud server in a log data processing system, wherein the log data processing system also includes an application server running an application service; the conversion device includes:

[0028] An acquisition module is used to acquire the original data generated by the target application server running the target application service;

[0029] A query module, configured to query a target log data processing policy corresponding to the target application service based on a pre-stored correspondence between the application service and the log data processing policy;

[0030] The conversion module is used to convert the original data into log data according to the target log data processing strategy.

[0031] In one embodiment, the acquisition module includes a receiving unit and a decompression unit, wherein:

[0032] The receiving unit is configured to receive compressed data sent by a target application server; the compressed data is obtained by compressing original data generated by running a target application service by the target application server;

[0033] The decompression unit is used to decompress the compressed data to obtain original data generated by the target server running the target application service.

[0034] In one embodiment, the conversion module includes a determination unit, a selection unit, and a calling unit, wherein:

[0035] The determining unit is configured to determine a target conversion type corresponding to the original data according to the log data processing strategy;

[0036] The selection unit is configured to determine, from pre-created process groups, a plurality of process groups corresponding to the target conversion type, and select a target process group whose working state is a load-unsaturated state from the plurality of process groups;

[0037] The calling unit is used to call the target process group to convert the original data into log data.

[0038] In one embodiment, the log data processing device further includes:

[0039] A status query module is used to query the working status of multiple process groups corresponding to the target conversion type;

[0040] A creation module is used to create a process group with a conversion type of the target conversion type when the working states of multiple process groups corresponding to the target conversion type are all in a load oversaturation state.

[0041] In one embodiment, the log data processing device further includes:

[0042] An instruction sending module is used to determine the application service corresponding to the target conversion type when the working states of multiple process groups corresponding to the target conversion type are all in an overloaded state, and send a flow limiting instruction to the server where the application service is located; the flow limiting instruction is used to instruct the server where the application service is located to reduce the rate of sending original data.

[0043] In one embodiment, the log data processing device further includes:

[0044] A deletion module is used to delete at least one process group in the load-unsaturated state among the multiple process groups corresponding to the target conversion type when the number of process groups in the load-unsaturated state reaches a preset threshold.

[0045] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0046] Obtain the original data generated by the target application server running the target application service;

[0047] According to the pre-stored correspondence between application services and log data processing strategies, query the target log data processing strategy corresponding to the target application service;

[0048] The original data is converted into log data according to the target log data processing strategy.

[0049] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0050] Obtain the original data generated by the target application server running the target application service;

[0051] According to the pre-stored correspondence between application services and log data processing strategies, query the target log data processing strategy corresponding to the target application service;

[0052] The original data is converted into log data according to the target log data processing strategy.

[0053] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0054] Obtain the original data generated by the target application server running the target application service;

[0055] According to the pre-stored correspondence between application services and log data processing strategies, query the target log data processing strategy corresponding to the target application service;

[0056] The original data is converted into log data according to the target log data processing strategy.

[0057] The present application discloses a log data processing method, which is applied to a log processing cloud server in a log data processing system. The log data processing system also includes an application server running an application service. After the application server runs the application service and generates raw data, the application server sends the collected raw data to the log processing cloud server. After the log processing cloud server obtains the raw data sent by the target application server, it queries the target log data processing policy corresponding to the target application service based on the correspondence between the pre-stored configured application service and the log data processing policy, and converts the raw data into log data based on the target log data processing policy.

[0058] The present application provides a log data processing method, apparatus, computer equipment, storage medium, and computer program product, which separate the log data processing service and set up a separate log processing cloud server to convert the original data into log data. On the one hand, this reduces the preemption of application service resources by the log data service, and on the other hand, ensures the processing efficiency of the log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a structural diagram of a log data processing system according to one embodiment of the present application;

[0060] Figure 2 This is a flow chart of a log data processing method according to one embodiment of the present application;

[0061] Figure 3 This is a schematic diagram of the deployment structure of a log processing cloud server according to an embodiment of the present application;

[0062] Figure 4 This is a structural block diagram of a log data processing device according to an embodiment of the present application;

[0063] Figure 5 This is a diagram of the internal structure of a computer device according to one embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0065] In related technologies, log collection services are typically deployed as standalone services on application servers. The collection, processing, and output of log data are all performed on the application server. For example, many log collection tools, such as Fluent and Logstash, typically read raw data, perform data processing through parsing and filtering components, and finally send the data to a designated terminal or database based on the required delivery requirements.

[0066] Because log data processing and application services are deployed simultaneously on application servers, log data processing can lead to resource congestion on the application servers as the volume of application services increases. Consequently, most solutions in the related art utilize a delayed processing mechanism, delaying log data processing until application server resources are sufficient. However, this can lead to delayed log data processing, which is unacceptable for applications that require timely log review or monitoring data display.

[0067] Based on this, the present application proposes a log data processing method, which is applied to a log processing cloud server in a log data processing system, wherein the log data processing system also includes an application server running an application service. After the application server runs the application service and generates raw data, the application server sends the collected raw data to the log processing cloud server. After the log processing cloud server obtains the raw data sent by the target application server, it queries the target log data processing policy corresponding to the target application service based on the correspondence between the pre-stored configured application service and the log data processing policy, and converts the raw data into log data according to the target log data processing policy.

[0068] Through the log data processing method provided in this application, the log data processing service is separated and processed, and a separate log processing cloud server is set up to convert the original data into log data. On the one hand, it reduces the preemption of application service resources by the log data service, and on the other hand, it ensures the processing efficiency of log data.

[0069] The present application provides a log data processing system and a log data processing method, which are applied to a log processing cloud server in the log data processing system, as well as corresponding devices, computer equipment, computer-readable storage media, and computer program products.

[0070] First, the log data processing system provided by this application is described in detail.

[0071] like Figure 1 FIG. 1 is a schematic diagram of the structure of the log data processing system of the present application, including a log processing cloud server and several application servers running different application services.

[0072] The log processing cloud server can be implemented as an independent server or a server cluster composed of multiple servers. The target application server can be implemented as an independent server or a server cluster composed of multiple servers, or can be implemented using a cloud server, which is not limited in this application.

[0073] In one embodiment, the target application server is used to collect raw data generated by running the target application service and send the raw data to the log processing cloud server.

[0074] In one embodiment, the log processing cloud server is used to obtain the original data generated by the target application server running the target application service, and query the target log data processing policy corresponding to the target application service based on the correspondence between the pre-stored configured application service and the log data processing policy, and finally convert the obtained original data into log data based on the queried target log data processing policy.

[0075] It should be noted that the application environment of the log data processing method of the present application is generally a cloud service environment. After the cloud provider deploys the log processing cloud server on the cloud, it performs relevant configurations on the log processing cloud server on the cloud according to the needs of the user, that is, the corresponding log data processing strategies are configured on the log processing cloud server for the original data corresponding to different application services and stored in the log processing cloud server.

[0076] In one embodiment, Figure 2 As shown, a log data processing method is provided, which is applied to Figure 1 The log processing cloud server in the log data processing system shown includes the following steps:

[0077] Step 201: Acquire original data generated by a target application server running a target application service.

[0078] Among them, in different application scenarios, the format and content of the required log data are different. The original data is the unprocessed data generated when the target application server runs the target application service. Corresponding to the log data in the expected form, the raw data is the data in the unexpected form. For example, in the e-commerce application scenario, the original data is the flow of each transaction. The transaction flow needs to be parsed, decoded, etc. to obtain the corresponding transaction log. For example, in the social application scenario, the original data is the operation process of different users. The operation process needs to be parsed, decoded, etc. to obtain the operation logs of different users.

[0079] In one embodiment, a collector for the target application service may be deployed on the target application server to collect raw data generated by the target application server running the target application service, and then send the data to the log processing cloud server.

[0080] In one embodiment, a collector for the target application service can also be deployed on the target application server to collect the original data generated by the target application server running the target application service, and then store it locally. The log processing cloud server periodically sends a request to the target application server to obtain the original data generated by the target application server running the target application service. After receiving the request, the target application server sends the original data generated by running the target application service stored locally to the log processing cloud server.

[0081] Step 203: Query the target log data processing policy corresponding to the target application service according to the pre-stored correspondence between the application service and the log data processing policy.

[0082] Among them, in different application scenarios, the content and form of the original data will be different, and the format and content of the processed log data will also be different. Therefore, the processing process of converting the original data into log data will be different.

[0083] In one embodiment, a log data processing cloud server can configure corresponding log data processing strategies for the raw data corresponding to different application services and store them in the log processing cloud server. After the log processing cloud server receives the raw data corresponding to the target application service, it queries the target log data processing strategy corresponding to the target application service based on the stored correspondence between the application service and the log data processing strategy. For example, in the log processing cloud server, the corresponding log data processing strategy configured for application service Service A is Strategy A, the corresponding log data processing strategy configured for application service Service B is Strategy B, and the corresponding log data processing strategy configured for application service Service C is Strategy C. When the log processing cloud server receives the raw data corresponding to Service B, that is, the target application service is Service B, then the query results in the target log data processing strategy being Strategy B.

[0084] The log data processing strategy includes the execution process of converting raw data into log data, including parsing, translation, transcoding, filtering, and sending. Parsing generally refers to the process of parsing raw data, translation and transcoding generally convert the parsed data into fixed-format data, filtering generally refers to the process of filtering the processed log data, and sending generally refers to sending the converted log data to a fixed terminal or database. The relevant art provides detailed descriptions of log data processing strategies for processing raw data to obtain corresponding log data. Please refer to the descriptions of log data processing strategies in the relevant art.

[0085] Step 205: Convert the original data into log data according to the target log data processing strategy.

[0086] In one embodiment, after the log processing cloud server determines the target log data processing strategy for the raw data, it converts the raw data into log data according to the log data processing strategy. For example, if the raw data is a plurality of files, the corresponding content needs to be read from the files and parsed, and the content in the files needs to be converted into the corresponding log data according to the format and content requirements of the log data. For example, if a file stores every operation process of a user, the log processing cloud server needs to read the content in the file and determine the location and content of each operation process in the file, and then form a corresponding operation log according to the format requirements of the operation log data.

[0087] In one embodiment, the log processing cloud server may pre-create a process group, and upon receiving raw data generated by running the target application service from the target application server, call the process group to process the received raw data to obtain log data corresponding to the raw data.

[0088] In one embodiment, after the log processing cloud server converts the original data into log data, it can store the original data in a database connected to the log processing cloud server for relevant personnel to query and view. It can also send the original data to the database connected to the application server so that relevant personnel can directly access the database connected to the application service to query the corresponding log data.

[0089] In this embodiment, all the original data generated by the target application service are sent to the log processing cloud server, which completes the conversion of the original data into log data, reducing the occupation of the target application service resources by the log data processing service, and ensuring the efficiency of the conversion of original data to log data when the application service is busy, so that relevant personnel can obtain the log data converted from the original data in a timely manner.

[0090] In one embodiment, step 201 specifically includes:

[0091] Step 201A: Receive compressed data sent by the target application server.

[0092] The compressed data is obtained by compressing the original data generated by running the target application service by the target application server.

[0093] In one embodiment, a collector for collecting raw data can be deployed on the target application server. The target application server generates raw data when running the target application service. The collector deployed on the target application server collects the raw data and, after a preset period of time, compresses the collected raw data to generate compressed data, which is then sent to the log processing cloud server.

[0094] Among them, when compressing the original data, a compression algorithm in the related technology can be used, such as the commonly used compression algorithms such as zip, rar, etc., which are not limited in this application.

[0095] Step 201B: decompress the compressed data to obtain the original data generated by the target server running the target application service.

[0096] In one embodiment, after the log processing cloud server receives the compressed data sent by the target application server, it determines the decompression algorithm corresponding to the compressed data, such as reading the compression algorithm flag in the compressed data to determine the corresponding decompression algorithm, and then uses the decompression algorithm to decompress the compressed data to obtain the original data generated by the target server running the target application service.

[0097] The decompression process corresponds to the compression process, that is, the decompression algorithm corresponds to the compression algorithm, and the compressed data needs to be decompressed using the decompression algorithm corresponding to the compression algorithm.

[0098] In this embodiment, the bandwidth resource occupation of the application server is reduced as much as possible, and the impact of the log data processing service on the application service is reduced.

[0099] In one embodiment, the log processing cloud server may pre-create process groups of different conversion types, and at least one process group of each conversion type is created. Upon receiving raw data generated by running the target application service from the target application server, the cloud server determines the conversion type of the raw data, and then calls the process group of the corresponding conversion type to process the received raw data to obtain log data corresponding to the raw data. Therefore, step 205 may specifically include:

[0100] Step 205A: Determine the target conversion type corresponding to the original data according to the log data processing strategy.

[0101] Among them, the conversion type refers to the different conversion processes from raw data to log data. For example, the process of converting transaction flow into transaction log is different from the process of converting operation flow into operation log, such as the above-mentioned parsing, translation, transcoding, and filtering processes. On the one hand, it is not necessary to execute all processes to convert raw data into log data. According to the different raw data and different user needs, some processes are selected for execution. For example, the conversion process of converting raw data into log data may not include filtering. On the other hand, for raw data from different application servers, the parsing, translation, transcoding and other processes will also be different. For example, some application servers send raw data in the form of files, and some application servers send raw data in the form of data streams, which are not organized in the form of files.

[0102] In one embodiment, after the log processing cloud server determines the log data processing strategy for the original data, it determines the target conversion type corresponding to the original data based on the processes that need to be performed in the conversion process from the original data to the log data in the strategy, such as whether parsing, translation, transcoding, filtering, etc. are required, as well as the specific processes of parsing, translation, transcoding, and filtering.

[0103] Step 205B: Determine multiple process groups corresponding to the target conversion type in the pre-created process groups, and select a target process group whose working state is a load-unsaturated state from the multiple process groups.

[0104] In one embodiment, after the log processing cloud server determines the target conversion type corresponding to the original data, it determines the process group whose conversion type is the target conversion type. If there is only one process group of the target conversion type, the process group is directly called to convert the original data into log data.

[0105] In one embodiment, after the log processing cloud server determines the target conversion type corresponding to the original data, it determines the process group whose conversion type is the target conversion type. If there are multiple process groups of the target conversion type, it determines the working status of the multiple process groups corresponding to the target conversion type, and selects the target process group with a working status of unsaturated load as the process group that converts the original data into log data.

[0106] Step 205C: Call the target process group to convert the original data into log data.

[0107] In one embodiment, after the log processing cloud server selects a target process group whose working state is an unsaturated load state, it calls the target process group to convert the original data into log data.

[0108] In one embodiment, the log processing cloud server can pre-create a process group for obtaining raw data. The process group is used to receive the raw data sent by the application server. When the log processing cloud server determines the target process group for converting the raw data into log data, the process group for obtaining the raw data sends the raw data to the target process group.

[0109] In this embodiment, corresponding process groups are configured for different conversion strategies. For the original data of the same log data processing strategy, the same process group can be reused to improve resource utilization.

[0110] In one embodiment, a process group includes multiple processes, each process is used to execute an execution process in the conversion process from original data to target data. For example, a process group includes four processes, which are used to execute parsing, translation, transcoding, and filtering processes respectively.

[0111] Among them, each process in the process group has an execution order. For example, after the process corresponding to parsing is completed, the parsed data is sent to the process corresponding to translation. After the process corresponding to translation is completed, the translated data is sent to the process corresponding to transcoding. After the process corresponding to transcoding is completed, the transcoded data is sent to the process corresponding to filtering. After the process corresponding to filtering is completed, the final log data is obtained, and the conversion of original data to log data is completed.

[0112] In one embodiment, the method further includes:

[0113] Step 207: Query the working status of multiple process groups corresponding to the target conversion type.

[0114] In one embodiment, the log processing cloud server may determine the process group whose conversion type is the target conversion type after a fixed period of time, and query the working status of each process group corresponding to the target conversion type.

[0115] In one embodiment, the log processing cloud server can query the CPU occupancy rate, memory occupancy rate, disk read and write rate, etc. of the process group, and determine the working status of the process group based on the CPU occupancy rate, memory occupancy rate, disk read and write rate, etc. of the process group.

[0116] Step 209: When the working states of multiple process groups corresponding to the target conversion type are all in a load oversaturation state, a process group whose conversion type is the target conversion type is created.

[0117] In one embodiment, the log processing cloud server queries the CPU occupancy rate, memory occupancy rate, disk read and write rate, etc. of the process group to determine whether the CPU occupancy rate, memory occupancy rate, disk read and write rate, etc. of the process group have reached the corresponding threshold value. If the corresponding domain value is reached, it is determined that the working status of the process group is in an overloaded state, and a process group with a conversion type of the target conversion type is created to alleviate the load pressure of process groups of other target conversion types.

[0118] In this embodiment, when the working statuses of all process groups meet the overload condition, it indicates that the existing process groups are insufficient to process the original data received at this time, and a new process group is created.

[0119] In one embodiment, when the working states of the process groups corresponding to the target conversion type are all in an overloaded state, on the one hand, the number of process groups of the target conversion type is increased, and on the other hand, the amount of raw data that the process groups of the target conversion type need to process can be reduced. Therefore, the above method can further include:

[0120] Step 211: When the working states of multiple process groups corresponding to the target conversion type are all in an overloaded state, determine the application service corresponding to the target conversion type, and send a current limiting instruction to the server where the application service is located.

[0121] The rate limiting instruction instructs the server hosting the application service to reduce the rate at which it sends raw data. Conversion types correspond to log data processing policies one-to-one. Each application service is configured with a corresponding log data processing policy. Based on the correspondence between conversion types and log data processing policies, and between log data processing policies and application services, the correspondence between conversion types and application services is determined, thereby identifying the application service corresponding to the target conversion type.

[0122] In one embodiment, when the log processing cloud server determines that the working status of each process group corresponding to the target conversion type meets the overload condition, the log processing cloud server determines the application service corresponding to the target conversion type based on the correspondence between the conversion type and the log data processing policy, and the correspondence between the log data processing policy and the application service, and sends a flow limiting instruction to the server where the application service is located. When the log processing cloud server determines that the working status of each process group corresponding to the target conversion type is good, that is, when the working status of each process group corresponding to the target conversion type is in an unsaturated load state, the log processing cloud server determines the application service corresponding to the target conversion type and sends a release flow limiting instruction to the server where the application service is located, for instructing the server where the application service is located to resume the rate of sending original data.

[0123] In one embodiment, upon receiving a throttling instruction, the server hosting the application service can store the collected raw data locally, reducing the rate at which the raw data is sent. Upon receiving a throttling release instruction from the log service, the rate at which the raw data is sent is restored, and the locally stored raw data is sent to the log processing cloud server. Newly collected raw data is not stored locally but is sent directly to the log processing cloud server.

[0124] In this embodiment, when the working status of the process groups all meet the overload condition, it means that the existing process groups at this time are insufficient to process the original data received at this time, thereby reducing the acquisition rate of the original data.

[0125] In one embodiment, the method further includes:

[0126] Step 213: When the number of process groups in the multiple process groups corresponding to the target conversion type whose working state is the load unsaturated state reaches a preset threshold, delete at least one process group in the multiple process groups corresponding to the target conversion type that is in the load unsaturated state.

[0127] Among them, the load unsaturation state corresponds to the load saturation state of the process group. The load saturation state is also called the full load state. For example, when the process group is in the load saturation state (full load state), the CPU occupancy rate, memory occupancy rate, disk read and write rate, etc. of the process group will have corresponding thresholds. When the corresponding threshold is exceeded, the process group is in an overload state and meets the overload condition. When it is lower than the corresponding threshold, the process group is in an unsaturated load state.

[0128] In one embodiment, the log processing cloud server queries the CPU occupancy rate, memory occupancy rate, disk read and write rate, etc. of the process group to determine whether the process group is in a load-unsaturated state. If the number of process groups corresponding to the target conversion type whose working status is a load-unsaturated state reaches a preset threshold, it means that there are too many process groups with the conversion type of the target conversion type at this time, and at least one process group in the load-unsaturated state among the multiple process groups corresponding to the target conversion type is deleted.

[0129] In this embodiment, when the number of process groups whose working status is a load-unsaturated state reaches a preset threshold, it means that there are too many process groups of this conversion type at this time, and the number of process groups of this conversion type is reduced to reduce the idle rate of process groups of this conversion type.

[0130] Next, a specific embodiment provided by this application is described in detail. Figure 3 FIG2 is a diagram showing the deployment structure of a log data processing service in a log processing cloud server according to an embodiment of the present application, including a data collection layer, a data scheduling layer, an execution layer, a resource monitoring layer, and a control layer.

[0131] Among them, the data collection layer includes multiple collectors, which are deployed in each application server. They read and compress the original data generated by the application server running the application service, and send the compressed original data to the data scheduling layer after compression is completed.

[0132] After receiving the compressed raw data sent by the application server, the data scheduling layer first decompresses it and sends the raw data to the corresponding process group according to the log data processing strategy for each application service issued by the control layer.

[0133] The execution layer consists of multiple process instances with different functions and types. It responds to requests from the data scheduling layer and provides processes for various raw data processing requirements, such as parsing, translation, transcoding, filtering, and delivery. The execution layer can also expand or shrink process groups, adding or removing them, based on instructions from the control layer.

[0134] The resource monitoring layer is used to periodically obtain corresponding resource usage data from the execution layer, such as the CPU usage, memory usage, disk read and write rates, network bandwidth, and number of connections of the process group.

[0135] The controller layer configures the log data processing strategy for each application service based on its log data processing requirements. This strategy is then sent to the data collection layer, which then sends the raw data to the corresponding process group according to the log data processing strategy issued by the controller for each application service. The resource monitoring layer determines whether the execution layer's process groups need to be expanded or reduced based on resource usage data collected from the execution layer. When this is necessary, the layer issues the appropriate instructions.

[0136] The execution process of the log processing cloud server is generally as follows:

[0137] 1. After receiving log data, the data scheduling layer determines the log data processing strategy corresponding to the raw data received and whether a process group of the conversion type corresponding to the log data processing strategy exists. If not, the controller is informed to start the corresponding process group of the conversion type. At the same time, the controller sends an instruction to the collector corresponding to the raw data to suspend data collection to avoid data congestion. After the process group of the conversion type corresponding to the log data processing strategy is started, the controller is sent an instruction to resume data collection.

[0138] 2. If it is determined that a process group of the conversion type corresponding to the log data processing strategy exists, the original data is forwarded to the load balancer corresponding to the process group of the conversion type, and the load balancer performs further data processing.

[0139] 3. The load balancer forwards the original data to the process group with the best load condition based on the working status of the process group of the conversion type, so that the corresponding process group converts the original data into log data.

[0140] 4. The controller communicates with each process group regularly. If a process group does not respond, it deletes the process group and creates a new process group to replace the deleted process group.

[0141] 5. The controller determines the working status of each process group of the execution layer based on the corresponding resource usage data of the execution layer obtained by the resource monitoring layer, determines whether each process group needs to be expanded or reduced, and issues corresponding instructions to the execution layer when expansion or reduction is required.

[0142] 6. The controller determines the working status of each process group in the execution layer based on resource usage data obtained by the resource monitoring layer. If the working status is poor, it issues a capacity expansion instruction to the execution layer and a flow control instruction to the corresponding collector in the collection layer to reduce the raw data collection rate. If too many process groups are underloaded, the controller issues a process group deletion instruction to the executor layer, i.e., a capacity reduction instruction.

[0143] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0144] The present application also provides a log data processing system, wherein the log data processing system includes at least an application server running an application service and a log processing cloud server. The application server is configured to collect raw data generated by the application service and send the collected raw data to the log processing cloud server. The log processing cloud server is configured to obtain the raw data generated by the application server running the application service, query the target log data processing policy corresponding to the target application service based on a pre-stored correspondence between the application service and the log data processing policy, and convert the raw data into log data based on the target log data processing policy.

[0145] The execution steps, specific limitations and descriptions of the log processing cloud server in the log data processing system provided in this application can be found in the above limitations on the log data processing method executed by the log processing cloud server, which will not be repeated here.

[0146] Based on the same inventive concept, the embodiments of the present application also provide a log data processing device for implementing the log data processing method involved above. The implementation solution provided by this device is similar to the implementation solution described in the above method, so the specific limitations of one or more log data processing device embodiments provided below can be found in the above limitations of the log data processing method and will not be repeated here.

[0147] In one embodiment, Figure 4 As shown, a log data processing device 400 is provided, which is applied to a log processing cloud server in a log data processing system. The log data processing system also includes an application server running an application service. The conversion device includes:

[0148] Acquisition module 401, used to acquire original data generated by the target application server running the target application service;

[0149] A query module 403 is configured to query a target log data processing policy corresponding to the target application service based on a pre-stored correspondence between application services and log data processing policies;

[0150] The conversion module 405 is configured to convert the original data into log data according to the target log data processing strategy.

[0151] In one embodiment, the acquisition module 401 includes a receiving unit 4011 (not shown in the figure) and a decompression unit 4013 (not shown in the figure), wherein:

[0152] The receiving unit 4011 is configured to receive compressed data sent by a target application server; the compressed data is obtained by the target application server compressing original data generated by running the target application service;

[0153] The decompression unit 4013 is used to decompress the compressed data to obtain the original data generated by the target server running the target application service.

[0154] In one embodiment, the conversion module 405 includes a determination unit 4051 (not shown in the figure), a selection unit 4053 (not shown in the figure), and a calling unit 4055 (not shown in the figure), wherein:

[0155] The determining unit 4051 is configured to determine a target conversion type corresponding to the original data according to the log data processing strategy;

[0156] The selection unit 4053 is configured to determine, from among the pre-created process groups, a plurality of process groups corresponding to the target conversion type, and select a target process group whose working state is an unsaturated load state from among the plurality of process groups;

[0157] The calling unit 4055 is used to call the target process group to convert the original data into log data.

[0158] In one embodiment, the log data processing device further includes:

[0159] A status query module 407 (not shown in the figure) is used to query the working status of multiple process groups corresponding to the target conversion type;

[0160] The creation module 409 (not shown in the figure) is used to create a process group with a conversion type of the target conversion type when the working states of multiple process groups corresponding to the target conversion type are all in a load oversaturation state.

[0161] In one embodiment, the log data processing device further includes:

[0162] The instruction sending module 411 (not shown in the figure) is used to determine the application service corresponding to the target conversion type when the working states of multiple process groups corresponding to the target conversion type are all in an overloaded state, and send a current limiting instruction to the server where the application service is located; the current limiting instruction is used to instruct the server where the application service is located to reduce the rate of sending original data.

[0163] In one embodiment, the log data processing device further includes:

[0164] The deletion module 413 (not shown in the figure) is used to delete at least one process group in the multiple process groups corresponding to the target conversion type that is in the unsaturated load state when the number of process groups in the unsaturated load state reaches a preset threshold.

[0165] Each module in the above-mentioned log data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0166] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store log data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a log data processing method is implemented.

[0167] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0168] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0169] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0170] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0172] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0173] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A log data processing method, characterized in that: The method is applied to a log processing cloud server in a log data processing system, wherein the log data processing system further includes an application server running an application service; the method includes: Obtain the original data generated by the target application server running the target application service; According to the pre-stored correspondence between application services and log data processing strategies, query the target log data processing strategy corresponding to the target application service; Converting the original data into log data according to the target log data processing strategy; wherein converting the original data into log data according to the target log data processing strategy includes: Determining a target conversion type corresponding to the original data according to the target log data processing strategy; Determine, from pre-created process groups, a plurality of process groups corresponding to the target conversion type, and select, from the plurality of process groups, a target process group whose working state is a load-unsaturated state; The target process group is called to convert the original data into log data.

2. The method according to claim 1, characterized in that The obtaining of original data generated by the target application server running the target application service includes: Receiving compressed data sent by a target application server; the compressed data is obtained by the target application server compressing original data generated by running the target application service; The compressed data is decompressed to obtain original data generated by the target application server running the target application service.

3. The method according to claim 1, characterized in that The method further comprises: Query the working status of multiple process groups corresponding to the target conversion type; When the working states of multiple process groups corresponding to the target conversion type are all in a load oversaturation state, a process group with a conversion type of the target conversion type is created.

4. The method according to claim 3, characterized in that The method further comprises: When the working states of multiple process groups corresponding to the target conversion type are all in an overloaded state, the application service corresponding to the target conversion type is determined, and a flow limiting instruction is sent to the server where the application service is located; the flow limiting instruction is used to instruct the server where the application service is located to reduce the rate of sending original data.

5. The method according to claim 3, characterized in that The method further comprises: When the number of process groups in a load-unsaturated state among the multiple process groups corresponding to the target conversion type reaches a preset threshold, at least one process group in a load-unsaturated state among the multiple process groups corresponding to the target conversion type is deleted.

6. A log data processing system, characterized in that: The log data processing system is used to implement the steps of the method according to any one of claims 1 to 5, and the log data processing system includes at least an application server running an application service and a log processing cloud server; the system includes: The application server is used to collect raw data generated by running application services and send the raw data to the log processing cloud server; The log processing cloud server is configured to obtain raw data generated by the application server running the application service; query a target log data processing policy corresponding to the target application service based on a pre-stored correspondence between the application service and the log data processing policy; and convert the raw data into log data based on the target log data processing policy; wherein converting the raw data into log data based on the target log data processing policy includes: Determining a target conversion type corresponding to the original data according to the target log data processing strategy; Determine, from pre-created process groups, a plurality of process groups corresponding to the target conversion type, and select, from the plurality of process groups, a target process group whose working state is a load-unsaturated state; The target process group is called to convert the original data into log data.

7. A log data processing device, characterized in that: The device is applied to a log processing cloud server in a log data processing system, wherein the log data processing system further includes an application server running an application service; the device includes: An acquisition module is used to acquire the original data generated by the target application server running the target application service; A query module, configured to query a target log data processing policy corresponding to the target application service based on a pre-stored correspondence between the application service and the log data processing policy; A conversion module is configured to convert the original data into log data according to the target log data processing strategy; wherein converting the original data into log data according to the target log data processing strategy includes: Determining a target conversion type corresponding to the original data according to the log data processing strategy; Determine, from pre-created process groups, a plurality of process groups corresponding to the target conversion type, and select, from the plurality of process groups, a target process group whose working state is a load-unsaturated state; The target process group is called to convert the original data into log data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images