Log data processing method, device, system and equipment and storage medium

Abstract

The invention provides a log data processing method, device, system and equipment and a storage medium, and relates to technologies such as log data processing and network access security and reliability of a distributed system. The log data processing method comprises the steps that an access request for an Nginx service is responded, a request processing flow is executed through a target work process, and the request processing flow comprises the steps that the access request is processed, log data corresponding to the access request is generated in the access request processing process, and the log data is transmitted to an isolation process; a log isolation transmission process is executed through the isolation process, the log isolation transmission process comprises writing the log data into a target pipeline corresponding to the log data, in the log data transmission process, the log data which cannot be written into the target pipeline is discarded, and the target pipeline is used for a log service system to read the log data. Therefore, the engineering process and the log service system are decoupled, and the situation that the Nginx service is normal but cannot be used is avoided.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and specifically to technologies such as log data processing of distributed systems, secure and reliable network access, etc., which are applied to log processing of NginX services, and in particular to a log data processing method, device, system, equipment and storage medium. Background Art

[0002] With the rapid development of Internet business, CDN (Content Delivery Network) technology plays an important role in improving network access speed and reliability, so ensuring the high availability of CDN system has become a focus of attention. In CDN system, log is an important part of operation and management service, so the generation and preservation of log is an important task of CDN system.

[0003] In the related art, the logs in the CDN system are generated by the Nginx service process and forwarded to the central management end through a dedicated log service system. However, in some scenarios, the Nginx service is normal but unavailable, which has a serious impact on the user experience. Summary of the invention

[0004] The present invention provides a log data processing method, apparatus, system, device and storage medium.

[0005] According to a first aspect of the present disclosure, a method for processing log data is provided, comprising: responding to an access request to an Nginx service, executing a request processing process through a target working process, the request processing process comprising processing the access request, generating log data corresponding to the access request during the processing of the access request, and transmitting the log data to an isolation process;

[0006] The log isolation transmission process is executed through the isolation process, and the log isolation transmission process includes writing the log data into the target pipe corresponding to the log data, wherein during the log data transmission process, the log data that cannot be written into the target pipe is discarded, and the target pipe is used for the log service system to read the log data.

[0007] According to a second aspect of the present disclosure, there is provided a log data processing device, comprising:

[0008] A work process execution unit, configured to execute a request processing flow in response to an access request to an Nginx service, wherein the request processing flow includes processing the access request, generating log data corresponding to the access request during the processing of the access request, and transmitting the log data to the isolation process;

[0009] The isolation process execution unit is used to execute the log isolation transmission process, which includes writing the log data into a target pipe corresponding to the log data, wherein during the log data transmission process, the log data that cannot be written into the target pipe is discarded, and the target pipe is used for the log service system to read the log data.

[0010] According to a third aspect of the present disclosure, there is provided a log data processing system, comprising: an Nginx node, a log service system, and a plurality of pipelines, wherein the log service system is used for communication with the Nginx node through the plurality of pipelines;

[0011] The Nginx node is used to respond to the access request to the Nginx service, and execute the request processing process through the target working process, the request processing process includes processing the access request, generating the log data corresponding to the access request in the process of processing the access request, and transmitting the log data to the isolation process; the log isolation transmission process is executed through the isolation process, the log isolation transmission process includes writing the log data into the target pipeline corresponding to the log data, wherein, during the log data transmission process, the log data that cannot be written into the target pipeline is discarded, and the target pipeline is used for the log service system to read the log data;

[0012] The log service system is used to read the log data through the target pipeline and manage and analyze the log data.

[0013] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the log data processing method described in the first aspect.

[0014] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the log data processing method described in the first aspect.

[0015] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the log data processing method described in the first aspect.

[0016] According to the technical solution provided by the present disclosure, an isolation process is added under the Nginx service, and the isolation process communicates with the log service system, and is used to transmit the log data generated by the working process to the log service system, so that the working process is isolated from the log service system. When the log service system cannot read the log normally due to an abnormality, on the one hand, the log service system and the working process are isolated by the isolation process, and the working process does not stop generating logs because the log service system cannot read the log normally; on the other hand, in the process of the isolation process transmitting the log data, the log data that cannot be written into the pipeline is discarded, so the working process can continue to generate logs, and the working process is decoupled from the log service system. Even if the log service system fails, the Nginx service process can still operate normally, improve the performance, reliability and robustness of the log processing system, and ensure that CDN users obtain the expected service quality.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram of Nginx service according to the first embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram of the interaction between the working process and the isolation process according to the first embodiment of the present disclosure;

[0022] Figure 4 is a schematic diagram according to a second embodiment of the present disclosure;

[0023] Figure 5 is a schematic diagram according to a third embodiment of the present disclosure;

[0024] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0025] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0026] Figure 8 FIG. 8 is a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] With the rapid development of Internet business, CDN technology plays an important role in improving network access speed and reliability, so ensuring the high availability of CDN services has become a focus of attention. In the CDN system, logs are an important part of operation and management services, so the generation and preservation of logs is an important task of the CDN system.

[0029] In the related art, the logs in the CDN system are generated by the Nginx service process and forwarded to the central management end through a dedicated log service system. However, in some scenarios, there are situations where the Nginx service is normal but unavailable, which has a serious impact on the user experience. For example, an abnormality in the log service system affects the normal availability of the Nginx service.

[0030] To solve the above problems, the present disclosure provides a method, device, system, equipment and storage medium for processing log data, which are related to log data processing of CDN system, network access security and reliability technology, etc. In the present disclosure, based on adding an isolation process in the Nginx service, it is used to isolate the working process and log service system of the Nginx service, so that the log service system and the working process can be decoupled. Specifically, the working process of the Nginx service transmits the generated log data to the isolation process, the isolation process of the Nginx service provides a log data processing function, writes the log data into the corresponding pipeline, and the log service system obtains the log data through the pipeline, and manages and analyzes the log data.

[0031] On the one hand, when the log service system is abnormal and stops obtaining log data from the pipeline, the worker process does not directly communicate with the pipeline, so the worker process will not receive the write failure signal fed back by the pipeline, and the worker process will continue to generate log data. Therefore, the isolation process isolates the log service system and the worker process to avoid the impact of the log service system, which causes the Nginx service to be normal but unavailable, ensuring that CDN users get the expected service quality. The Nginx service can discard the log data that cannot be written to the pipeline, thereby saving the sustainable transmission log data of the worker process.

[0032] On the other hand, the isolated process can write log data into the pipeline without changing the communication structure between the log service system and each service, thus ensuring the universality of the communication structure of the log service system.

[0033] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0034] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the log data processing method provided by the first embodiment of the present disclosure includes:

[0035] S101, responding to an access request to the Nginx service, executing a request processing flow through a target working process, the request processing flow including processing the access request, generating log data corresponding to the access request during the processing of the access request, and transmitting the log data to the isolation process;

[0036] The present disclosure is applied to Nginx nodes in CDN systems. Nginx nodes can be used as edge nodes of CDN to load cache and distribute static resources. When a user initiates an access request to a website or service through a browser or other client, the access request is sent to the Nginx node. The Nginx node responds to the access request to the Nginx service. When the Nginx node responds to the access request, it selects at least one working process to process the access request.

[0037] Among them, Figure 2 As shown in the figure, the Nginx service adopts the Master-Worker multi-process architecture, which includes a master process (Master Process) and one or more worker processes (Worker Processes). The master process is used to select at least one target worker process from multiple worker processes according to the configuration file of the Nginx service, and then the target worker process executes the request processing flow to complete the processing of the access request.

[0038] The target work process executes the request processing flow including processing the access request. In the process of processing the access request, the target work process generates log data corresponding to the access request. The log data needs to be transmitted to the log service system. The target work process can transmit the log data in the following ways:

[0039] In some implementations, the work process writes log data to local files, and the log service system periodically reads these files, obtains logs, and uploads the logs to the center for centralized processing. However, when writing logs to local files, frequent local disk I / O affects performance, and log uploads have a certain delay.

[0040] In some implementations, the work process writes logs to a pipeline (such as a pipe), and the log service system reads the logs from the pipeline in real time and directly forwards them to the center for processing. However, the log writing operation of the Nginx process and the reading operation of the log service system are synchronized. If the log service system works abnormally (failure occurs, etc.), it stops reading logs, and the pipe peer no longer receives data, resulting in the Nginx service being unable to write logs, and the Nginx service process being blocked, affecting the normal operation of the Nginx service process, that is, the Nginx service cannot be used in a normal state, affecting the normal use of users.

[0041] In some implementation methods, the working process of Nginx service directly sends log directly to the center side log service system through the syslog extension module. However, the log that the center side log service system will process is not only the log of the Nginx service process, but also the log of other services, and even if other services adopt the mode of channel to communicate with the log service system, the writing and reading of log are not required to be synchronous. If the syslog extension module is adopted to directly send log directly to the center side log service system, the communication structure of the center side log service system needs to be modified to adapt to the interface of the syslog extension module, causing the interface of other services to also need corresponding modification, which has a large impact on the whole log service system, making the log service system lack universality, and the syslog extension module is directly connected to the center side log service system, which has strong dependence on the network, needs extra network bandwidth, and easily loses log when the transmission is abnormal.

[0042] To this end, in some implementations, an isolation process is added to the Nginx service, and the isolation process is used to isolate the log data generated by the target working process for transmission, so as to isolate the target working process from the log service system. The specific implementation method is as shown in step S102.

[0043] S102, executing the log isolation transmission process through the isolation process, the log isolation transmission process includes writing the log data into the target pipe corresponding to the log data, wherein during the log data transmission process, the log data that cannot be written into the target pipe is discarded, and the target pipe is used for the log service system to read the log data.

[0044] In the present disclosure, the isolation process can process the log data of at least one target working process and transfer the log data to the target pipeline.

[0045] Optionally, the log isolation transmission process includes: obtaining log data, processing log data, determining the target pipeline corresponding to the log data, and writing the log data into the target pipeline. There are multiple pipelines between the Nginx service and the log service system, and different pipelines transmit different types of log data. Therefore, when processing log data, the target pipeline is determined according to the attributes of the log data such as type, source, etc., and then the log data is written into the corresponding target pipeline, and the log service system reads the log data from the target pipeline.

[0046] In this embodiment, since the log data of the target working process is transmitted to the log service system by the isolation process, when the log service system is abnormal, the target working process will not receive the write failure signal fed back by the pipeline, and the target working process will not stop generating log data, so the log service system and the working process are isolated based on the isolation process. As for the log data that cannot be written into the target pipeline, it will be discarded during the log data transmission process. The specific implementation method is as follows:

[0047] In an optional implementation, log data is transmitted between the working process and the isolation process based on a buffer. For example, the working process transmits the generated log data to the buffer, and the isolation process obtains the log data from the buffer, processes the log data, and writes it to the target pipeline. Among them, the buffer does not require synchronization of data reception and transmission, so when the data in the buffer is not sent, it does not affect the reception of data. Therefore, the target working process is not affected by abnormalities on the other end. Furthermore, if the log data cannot be written to the target pipeline, the log data in the buffer is discarded, so that the working process can continue to write log data to the buffer to ensure the normal operation of the working process.

[0048] In an optional implementation, log data is transmitted between the target working process and the isolated process based on a data transmission module, and the data transmission module includes a buffer. When the capacity of the buffer reaches the maximum capacity, the data transmission module discards the log data in the buffer. Therefore, when the isolated process stops reading the log data in the target pipeline based on the log service system, causing the isolated process to be unable to write log data to the target pipeline, the data transmission module discards the log data in the buffer, and the target working process can run normally, generate log data normally, and transfer the log data to the buffer, thereby ensuring the normal operation of the target working process.

[0049] Optionally, the data transmission module includes a UDP module or a socket module. The UDP module includes a receiving buffer for storing the arriving log data. When the buffer is full, the arriving log data will be discarded. Similarly, the socket module includes a receiving buffer and a sending buffer. When the capacity in the buffer reaches the maximum capacity, in non-blocking mode, the send call will immediately return an error and discard the log data. Based on the data transmission module, data transmission between the working process and the isolated process is realized, and the log data that cannot be written is processed to avoid affecting the normal operation of the working process. The structure is simple.

[0050] In an optional implementation, a log data discarding function can be added to the isolation process. After the isolation process writes the log data to the corresponding target pipeline, if a write failure signal is received from the target pipeline within a preset time interval, the log data will be discarded. That is, the isolation process discards the log data that cannot be written.

[0051] Therefore, during the log data transmission between the target working process and the isolated process, the log data that cannot be written to the target pipeline can be discarded, and the log data that cannot be written to the target pipeline can also be discarded during the isolated transmission of log data by the isolated process.

[0052] Optionally, the present disclosure can determine whether the log data is log data that cannot be written in the following manner, for example, in the isolation process executing the log isolation transmission process, after the log data is written to the target pipe corresponding to the log data, if a write failure signal is received within a preset time interval, it is determined that the log data cannot be written to the target pipe. Among them, when the log service system stops reading logs from the target pipe, the target pipe will return a write failure signal to the isolation process, so the isolation process can receive the write failure signal within the preset time interval.

[0053] Alternatively, if Figure 3 As shown in the figure, when the Nginx node responds to the access request, the interaction process between the target working process and the isolated process is as follows:

[0054] S301, receiving an access request;

[0055] The main process distributes the access request to each target working process. After receiving the access request, the target working process parses the access request and then processes the access request process.

[0056] S302, processing the access request, and generating log data during the process of processing the access request;

[0057] In the process of receiving and processing access requests, the working process will generate log data, which records the processing process of the access request. The log data includes access logs and error logs.

[0058] S303, transferring the log data to the buffer.

[0059] The target working process sends the generated log data to the buffer, which is then transmitted to the isolated process.

[0060] S304, obtaining log data from the buffer;

[0061] When the isolation process is working, it continuously obtains log data from the buffer to forward the log data to the log service system, and the log service system records the data generated by the entire access request.

[0062] S305, determining the target pipeline corresponding to the log data according to the attributes of the log data;

[0063] The isolation process and the log service system are connected through multiple pipelines. The isolation process needs to process the log data to determine the target pipeline to which the log data is to be written, so as to facilitate data classification management and storage.

[0064] S306, writing the log data into the target pipeline.

[0065] The isolated process writes the log data into the target pipeline and transmits the log data to the log service system through the target pipeline.

[0066] Through the above process, the isolation process realizes the transmission of the log data generated by the target working process to the log service system through isolated transmission, thereby realizing the decoupling of the log service system and the working process.

[0067] Exemplarily, the isolation process can be achieved by adding a syslog process in the Nginx service.

[0068] In this embodiment, an isolation process is added under the Nginx service, and the isolation process communicates with the log service system, and is used to transmit the log data generated by the working process to the log service system, so that the working process is isolated from the log service system. When the log service system cannot read the log normally due to an abnormality, on the one hand, the log service system and the working process are isolated by the isolation process, and the working process does not stop generating logs because the log service system cannot read the log normally; on the other hand, in the process of transmitting log data by the isolation process, the log data that cannot be written into the pipeline is discarded, so the working process can continue to generate logs, and the working process is decoupled from the log service system. Even if the log service system fails, the Nginx service process can still operate normally, improving the performance, reliability and robustness of the log processing system, and ensuring that CDN users obtain the expected service quality.

[0069] Figure 4 is a schematic diagram according to the second embodiment of the present disclosure. Figure 4 As shown, the log data processing method provided by the second embodiment of the present disclosure includes:

[0070] S401, when the Nginx service is restarted or reloaded, obtain the configuration file of the Nginx service;

[0071] The execution device of the embodiment of the present disclosure may be an Nginx node. Each time the Nginx service is restarted or reloaded, a new configuration file is obtained, which is used to configure the parameters of the sub-processes under the Nginx service, such as the main process, the working process, and the isolated process.

[0072] Optionally, the configuration file includes information indicating the number of work processes, environmental parameters of the work processes, etc.

[0073] S402, creating a working process and an isolation process corresponding to the working process according to the configuration file of the Nginx service;

[0074] Each time the Nginx service is restarted or reloaded, the Nginx service will recreate the working process and the isolated process, and the access request to the Nginx service will also be processed by the newly created working process.

[0075] If the Nginx service is restarted or reloaded during the process of responding to the access request, after the work process is created, the request processing flow is executed through the work process to continue processing the access request. At the same time, after the isolation process is created, the log isolation transmission flow is executed through the isolation process to transmit the log data generated by the work process to the log service system.

[0076] If the Nginx service does not respond to the access request, the Nginx service restarts or reloads, creates the working process and the isolation process, and waits for the access request to the Nginx service. The main process selects the working process to execute the request processing flow and processes the access request. The isolation process executes the log isolation transmission process and transmits the log data generated by the working process to the log service system.

[0077] Among them, the work process executes the request processing process and the isolation process executes the log isolation transmission process. The parameters mentioned above Figure 1 The embodiments are not described in detail here.

[0078] In this embodiment, when the Nginx service is restarted or reloaded, an isolation process is created while creating a new working process, and a new isolation process is configured for the new working process to transmit logs, so as to achieve decoupling of the working process from the log service system.

[0079] Figure 5 is a schematic diagram according to the third embodiment of the present disclosure. Figure 5 As shown, the log data processing method provided by the third embodiment of the present disclosure includes:

[0080] S501, when the Nginx service is restarted or reloaded, obtain the configuration file of the Nginx service;

[0081] S502, creating a working process and an isolation process corresponding to the working process according to the configuration file of the Nginx service;

[0082] The implementation principles and technical effects of S501 to S502 may refer to the aforementioned embodiments and will not be described in detail.

[0083] S503, determining the target communication port of the working process and the isolated process, where the target communication port is different from the old communication port before the Nginx service is restarted or reloaded;

[0084] Among them, the execution device of the embodiment of the present disclosure can be an Nginx node.

[0085] When the Nginx service is restarted or reloaded, the Nginx service creates a new working process and a new isolated process to run. The old working process and old isolated process that were running before the Nginx service is restarted or reloaded will stop running. During the switching process between the old and new processes, since the new process needs a certain amount of time to run stably, if the old process is stopped directly during the switching process between the old and new processes, the Nginx service will be interrupted, affecting user access to the Nginx service.

[0086] Therefore, after the new working process and the new isolated process are created, the old working process and the old isolated process will still run for a while. After the new working process and the new isolated process run stably, the old working process and the old isolated process will be stopped. The simultaneous operation of the new and old processes ensures that Nginx will not interrupt the current service when reloading the configuration, thus ensuring a good user experience.

[0087] In the disclosed embodiment, since an isolation process is newly added, and the log data generated by the working process is transmitted to the log service system through the isolation process, when the new and old working processes are running at the same time, if the log data of the new and old working processes are transmitted to the buffer through the same port, the new isolation process may obtain the log data of the old working process, and the old isolation process may obtain the log data of the new working process, resulting in the staggered exchange of the log data of the new and old processes, affecting the subsequent processing of the log data by the log service system.

[0088] Based on this, in order to avoid process interleaving and interaction and achieve smooth transition and seamless switching between new and old processes, the embodiment of the present disclosure configures the new and old processes to run on different communication ports. For example, the communication ports used by the newly created working process and the isolated process are different from the communication ports used by the old working process and the old isolated process.

[0089] In an optional implementation, a multi-port rotation mechanism is set up to rotate the communication ports so that the communication ports of the new working process and the new isolated process are different from the communication ports of the old working process and the old isolated process.

[0090] Optionally, the multi-communication port rotation mechanism refers to setting multiple groups of communication ports and setting rotation rules. When the Nginx service is restarted or reloaded, the rotation rules are used to determine the target communication ports for the currently created working process and the isolated process.

[0091] Exemplarily, port identifier rotation rules can be set to rotate communication ports. For example, each communication port is marked with a port identifier, and the order of port identifier replacement is set. When the Nginx service is restarted or reloaded, the execution device replaces the port identifier according to the port identifier replacement order. The replaced port identifier is the identifier of the communication port of the working process and the isolated process created after the Nginx service is restarted or reloaded. In this way, when creating a working process and an isolated process, the replaced port identifier can be obtained, and the target communication port of the working process and the isolated process can be configured.

[0092] For example, take three groups of communication ports as an example. The port identifiers corresponding to the three groups of communication ports are d1, d2, and d3, and the port identifier replacement order is d1-d2-d3-d1. When the Nginx service is started, the port identifier is d1. After the working process and the isolated process are created, the communication port corresponding to d1 is used as the communication port of the working process and the isolated process. After the Nginx service is restarted or reloaded, according to the port identifier replacement order, it is replaced with port identifier d2, and the working process and the isolated process are created, and the d2 port identifier replacement order is used. After the Nginx service is restarted or reloaded again, according to the port identifier replacement order, it is replaced with port identifier d3, and the working process and the isolated process are created, and the d3 port identifier replacement order is used. It can be seen that the communication port of each newly created working process and isolated process is different from the communication port of the working process and isolated process before the Nginx service is restarted or reloaded.

[0093] In this example, when the Nginx service is restarted or reloaded, the target communication port of the new working process and the new isolated process can be determined, and the target communication port is different from the communication port of the old working process and the old isolated process before the Nginx service is restarted or reloaded. Based on the port identification replacement method, the target communication port can be quickly located.

[0094] Alternatively, in other examples, the rotation of the communication ports is completed by taking the remainder of the number of communication ports by the number of times the Nginx service is restarted or reloaded. For example, each time the Nginx service is restarted or reloaded, the number of changes N is recorded, and N=i+1, where i≥0. The communication port to be rotated is determined based on the number of changes N and the preset number of communication ports; then the communication port to be rotated is configured as the target communication port.

[0095] Exemplarily, taking setting 3 groups of communication ports as an example, the communication ports to be rotated are determined by taking the remainder of the number of communication ports after the number of changes, wherein when the remainder is 0, the communication port is X1, when the remainder is 1, the communication port is X2, and when the remainder is 2, the communication port is X3.

[0096] When the Nginx service is started, the number of changes is 0, and the number of changes takes the remainder of the number of communication ports: 0%3=0, that is, the remainder is 0, and the X1 communication port is used. When the Nginx service is restarted or reloaded, the number of changes is recorded as 1, and the number of changes takes the remainder of the number of communication ports: 1%3=1, that is, the remainder is 1, and the X2 communication port is used. When the Nginx service is restarted or reloaded again, the number of changes is recorded as 2, and the number of changes takes the remainder of the number of communication ports: 2%3=2, that is, the remainder is 2, and the X3 communication port is used. When the Nginx service is restarted or reloaded again, the number of changes is recorded as 3, and the number of changes takes the remainder of the number of communication ports: 3%3=0, that is, the remainder is 0, and the X1 communication port is used, and each communication port is rotated in turn.

[0097] In this example, when the Nginx service is restarted or reloaded, the target communication port of the new working process and the new isolated process can be determined, and the target communication port is different from the communication port of the old working process and the old isolated process before the Nginx service is restarted or reloaded. Based on the number of changes and the method of taking the remainder of the number of communication ports by the number of changes, multi-port rotation is realized, and the method of recording the number of changes is simpler and more convenient.

[0098] S504, configure the target communication port for the working process and the isolated process.

[0099] Among them, the working process and isolation process of the target communication port are configured. When responding to the access request, the working process transmits the log data to the isolation process through the target communication port during the execution of the request processing process. The isolation process executes the log isolation transmission process and transmits the log data generated by the working process to the log service system.

[0100] Among them, the work process executes the request processing process and the isolation process executes the log isolation transmission process. The parameters mentioned above Figure 1 The embodiments are not described in detail here.

[0101] In the disclosed embodiment, the communication port refers to the communication port between the working process and the data transmission module (or buffer), and between the data transmission module (or buffer) and the isolation process. The communication port used between the working process and the data transmission module (or buffer) is the same as the communication port used between the data transmission module (or buffer) and the isolation process. In this way, the working process transfers the log data to the corresponding storage area through the target communication port, and the isolation process can read the log data in the storage area through the target communication port.

[0102] In the disclosed embodiment, after the Nginx service is restarted or reloaded, a new communication port is allocated to the newly created working process and isolated process. The new working process transmits log data to the buffer based on the new communication port. The new isolated process obtains the log data in the buffer through the new communication port without interleaving with the old working process, thereby achieving smooth transition and seamless switching between the old and new processes when the Nginx service is restarted or reloaded.

[0103] Figure 6 Schematic diagram of the fourth embodiment of the present disclosure. Figure 6 As shown, the log data processing device 600 provided in the fifth embodiment of the present disclosure includes:

[0104] The work process execution unit 601 is used to execute a request processing flow in response to an access request to the Nginx service, the request processing flow including processing the access request, generating log data corresponding to the access request during the processing of the access request, and transmitting the log data to the isolation process;

[0105] The isolation process execution unit 602 is used to execute the log isolation transmission process, which includes writing log data into the target pipe corresponding to the log data. During the log data transmission process, the log data that cannot be written into the target pipe is discarded, and the target pipe is used for the log service system to read the log data.

[0106] In some embodiments, a buffer is also included, and log data is transmitted between the working process execution unit 601 and the isolation process execution unit 602 through the buffer. During the transmission of the log data, the log data in the buffer that cannot be written into the target pipeline is discarded.

[0107] In some embodiments, a data transmission module (such as UDP in the figure) is also included, and the data transmission module includes a buffer.

[0108] In some embodiments, the data transmission module includes a UDP module or a socket module.

[0109] In some embodiments, the work process execution unit 601 includes a receiving module for receiving an access request; a first processing module for processing the access request; and a generating module for generating log data during the processing of the access request and transmitting the log data to a buffer.

[0110] In some embodiments, the isolation process execution unit 602 includes a first acquisition module for acquiring log data from a buffer; a second processing module for processing the log data and determining that the log data corresponds to a target pipeline based on the attributes of the log data; and a writing module for writing the log data into the target pipeline.

[0111] In some embodiments, the isolation process execution unit 602 also includes: a collection module (not shown in the figure), which is used to collect the write failure signal fed back by the target pipeline. When the collection module collects the write failure signal, the isolation process execution unit 602 determines that the log data cannot be written to the target pipeline. When the log service system stops reading logs from the target pipeline, the target pipeline returns a write failure signal to the isolation process.

[0112] In some embodiments, a work process execution unit 601 is also included. The work process execution unit 601 includes: a creation module for creating a work process and an isolation process. The work process is executed on the work process execution unit to implement execution service requests. The isolation process is executed on the isolation process execution unit to implement execution log data forwarding.

[0113] In some embodiments, the main process execution unit 603 also includes: a determination module, used to determine the target communication port of the working process and the isolated process, and the target communication port is different from the old communication port before the Nginx service is restarted or reloaded; a determination module, used to determine the target communication port of the working process and the isolated process, and the target communication port is different from the old communication port before the Nginx service is restarted or reloaded.

[0114] In some embodiments, the determination module is further used to determine the target communication ports of the working process and the isolated process based on a multi-communication port rotation mechanism.

[0115] In some embodiments, the determination module includes: a first update sub-module, used to replace the port identifier according to a preset port identifier change order when the Nginx service is restarted or reloaded, and each port identifier corresponds to a communication port; a first acquisition sub-module, used to obtain the port identifier; a first determination sub-module, used to determine the target communication port of the working process and the isolation process based on the port identifier.

[0116] In some embodiments, the determination module includes: a second update sub-unit, used to update the number of changes N when the Nginx service is restarted or reloaded, where N=i+1; a second acquisition sub-module, used to obtain the number of changes N; a second determination sub-module, determining the communication port to be rotated based on the number of changes N and the preset number of communication ports, and determining the communication port to be rotated as the target communication port.

[0117] Figure 6 The log data processing device provided can execute the steps involved in the terminal in the above-mentioned corresponding method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0118] Figure 7 Schematic diagram of the fifth embodiment of the present disclosure. Figure 7As shown, the log data processing system 700 provided by the fifth embodiment of the present disclosure includes: an Nginx node 701, a log service system 703 and a plurality of pipelines 702, the log service system 703 is used for communication with the Nginx node 701 through the plurality of pipelines 702;

[0119] Among them, Nginx node 701 is used to respond to the access request to the Nginx service, and execute the request processing process through the target working process, the request processing process includes processing the access request, generating log data corresponding to the access request during the processing of the access request, and transmitting the log data to the isolation process; the log isolation transmission process is executed through the isolation process, and the log isolation transmission process includes writing the log data into the target pipe 702 corresponding to the log data, wherein, during the log data transmission process, the log data that cannot be written into the target pipe 702 is discarded, and the target pipe 702 is used for the log service system 703 to read the log data;

[0120] The log service system 703 is used to read log data through the target pipeline and manage and analyze the log data.

[0121] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the solution provided by any of the above embodiments.

[0122] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a solution provided by any of the above embodiments.

[0123] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0124] According to an embodiment of the present disclosure, the present disclosure also provides a vehicle-mounted terminal, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the solution provided by any of the above embodiments.

[0125] According to an embodiment of the present disclosure, the present disclosure further provides a vehicle, which includes the vehicle-mounted terminal provided by the above embodiment.

[0126] Figure 8 800 is a schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0127] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can store data stored in a read-only memory (ROM) ( Figure 8 A computer program in ROM 802 is loaded from storage unit 808 to random access memory (RAM) ( Figure 8 The computer programs in the RAM 803 are used to perform various appropriate actions and processes. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. The input / output (I / O) interface ( Figure 8 The I / O interface 805 is also connected to the bus 804 .

[0128] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0129] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a processing method for log data. For example, in some embodiments, the processing method for log data may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the processing method for log data described above may be executed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the log data processing method in any other appropriate manner (for example, by means of firmware).

[0130] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable log data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0135] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0136] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0137] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for processing log data, comprising: In response to an access request to an Nginx service, executing a request processing flow through a target working process, the request processing flow including processing the access request, generating log data corresponding to the access request during the processing of the access request, and transmitting the log data to the isolation process; The log isolation transmission process is executed through the isolation process, and the log isolation transmission process includes writing the log data into the target pipe corresponding to the log data, wherein during the log data transmission process, the log data that cannot be written into the target pipe is discarded, and the target pipe is used for the log service system to read the log data.

2. The processing method according to claim 1, wherein: The log data is transmitted between the target working process and the isolated process based on a buffer. During the log data transmission process, the log data in the buffer that cannot be written into the target pipeline is discarded.

3. The processing method according to claim 2, wherein: The log data is transmitted between the target working process and the isolated process based on a data transmission module, the data transmission module includes the buffer, and the data transmission module discards the log data in the buffer when the capacity of the buffer reaches the maximum capacity.

4. The processing method according to claim 3, wherein: The data transmission module includes a UDP module or a socket module.

5. The processing method according to any one of claims 1 to 4, wherein: After the log data is written into the target pipeline corresponding to the log data, the method further includes: If a write failure signal is received within a preset time interval and it is determined that the log data cannot be written to the target pipeline, when the log service system stops reading logs from the target pipeline, the target pipeline returns a write failure signal to the isolation process.

6. The processing method according to any one of claims 1 to 5, wherein: include: When the Nginx service is restarted or reloaded, obtain the configuration file of the Nginx service; A working process and an isolation process corresponding to the working process are created according to the configuration file of the Nginx service, wherein the working process is used to execute the request processing process when responding to an access request to the Nginx service, and the isolation process is used to execute the log isolation transmission process.

7. The processing method according to claim 6, wherein: After creating the working process and the isolation process according to the configuration file of the Nginx service, the method further includes: Determine a target communication port between the working process and the isolated process, wherein the target communication port is different from an old communication port before the Nginx service is restarted or reloaded; The target communication port is configured for the working process and the isolation process, and the working process transmits the log data to the isolation process through the target communication port when executing the request processing flow.

8. The processing method according to claim 7, wherein: The determining of the target communication port of the working process and the isolated process includes: Based on a multi-communication port rotation mechanism, target communication ports of the working process and the isolated process are determined.

9. The processing method according to claim 8, wherein: The determining the target communication ports of the working process and the isolated process based on the multi-communication port rotation mechanism includes: Obtain port identifiers, each of which corresponds to a communication port. When the Nginx service is restarted or reloaded, the port identifiers are replaced according to a preset port identifier replacement sequence; The target communication ports of the working process and the isolated process are determined according to the port identifier.

10. The processing method according to claim 8, wherein: The determining the target communication ports of the working process and the isolated process based on the multi-communication port rotation mechanism includes: Obtain the number of changes N, and when the Nginx service is restarted or reloaded, update the number of changes N=i+1; The communication port to be rotated is determined according to the number of changes N and the preset number of communication ports, and the communication port to be rotated is determined as the target communication port.

11. A log data processing device, wherein: include: A work process execution unit, configured to execute a request processing flow in response to an access request to an Nginx service, wherein the request processing flow includes processing the access request, generating log data corresponding to the access request during the processing of the access request, and transmitting the log data to the isolation process; The isolation process execution unit is used to execute the log isolation transmission process, which includes writing the log data into a target pipe corresponding to the log data, wherein during the log data transmission process, the log data that cannot be written into the target pipe is discarded, and the target pipe is used for the log service system to read the log data.

12. A log data processing system, wherein: include: An Nginx node, a log service system, and a plurality of pipelines, wherein the log service system is used for communication with the Nginx node through the plurality of pipelines; The Nginx node is used to respond to the access request to the Nginx service, and execute the request processing process through the target working process, the request processing process includes processing the access request, generating the log data corresponding to the access request in the process of processing the access request, and transmitting the log data to the isolation process; the log isolation transmission process is executed through the isolation process, the log isolation transmission process includes writing the log data into the target pipeline corresponding to the log data, wherein, during the log data transmission process, the log data that cannot be written into the target pipeline is discarded, and the target pipeline is used for the log service system to read the log data; The log service system is used to read the log data through the target pipeline and manage and analyze the log data.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the processing method according to any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the processing method according to any one of claims 1 to 10.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the processing method according to any one of claims 1 to 10 are implemented.

Images