Log integration system, method and device of API gateway and storage medium

Abstract

The invention discloses a log integration system, method and device of an API gateway and a storage medium. The system comprises an API gateway, a distributed message service center cluster and a log processing module. The distributed message service center cluster is respectively connected with the API gateway and the log processing module; the API gateway is used for collecting the log data of the gateway, serializing the log data in combination with a log structure definition file, and sending the serialized log data to the distributed message service center cluster; the distributed message service center cluster is used for storing the serialized log data; and the log processing module is used for pulling the serialized log data from the distributed message service center cluster, and re-serializing the serialized log data in combination with the log structure definition file to obtain undefined-structure log data which can be used by a business application. According to the system, the log data collection performance and the log integration convenience are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of log processing technology, and in particular to a log integration system, method, device, and storage medium for an API gateway. Background Art

[0002] As digital transformation advances, Application Programming Interface (API) gateways play a key role in enterprise digital transformation, helping them achieve digital innovation, business optimization, and enhanced customer experience. By effectively leveraging the capabilities and advantages of API gateways, enterprises can better adapt to market changes, improve operational efficiency, and achieve sustained growth and competitive advantage. However, the explosive growth in the number of APIs and concurrent access volume has placed unprecedented performance pressure on API gateway applications, such as log collection and integration.

[0003] Traditional enterprises typically deploy API gateways across multiple clusters and nodes. Therefore, log collection requires centralized collection of proxy traffic logs from all nodes. Logs must be used by various business applications, such as security analysis, situational awareness, and monitoring and alerting. However, existing methods make it difficult to easily integrate API gateway traffic logs with other business applications. Summary of the Invention

[0004] The present invention provides a log integration system, method, device and storage medium for an API gateway to solve the problems of low log data collection performance and inconvenient integration in the prior art.

[0005] According to one aspect of the present invention, a log integration system for an API gateway is provided, the system comprising an API gateway, a distributed message service center cluster, and a log processing module; the distributed message service center cluster is connected to the API gateway and the log processing module respectively;

[0006] The API gateway is used to collect the log data of the gateway, serialize the log data in combination with the log structure definition file, and send the serialized log data to the distributed message service center cluster;

[0007] The distributed message service center cluster is used to store the serialized log data;

[0008] The log processing module is used to pull the serialized log data from the distributed message service center cluster, and re-serialize the serialized log data in combination with the log structure definition file to obtain log data with no defined structure that can be used by business applications.

[0009] According to another aspect of the present invention, a log integration method for an API gateway is provided, the method comprising:

[0010] Collect the gateway's log data through the API gateway, serialize the log data in combination with the log structure definition file, and send the serialized log data to the distributed message service center cluster;

[0011] The serialized log data is pulled from the distributed message service center cluster through a log processing module, and the serialized log data is reserialized in combination with the log structure definition file to obtain log data with no defined structure that can be used by business applications.

[0012] According to another aspect of the present invention, there is provided an electronic device, comprising: at least one processor; and

[0013] a memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log integration method of the API gateway described in any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log integration method of the API gateway described in any embodiment of the present invention when executed.

[0016] The embodiment of the present invention provides a log integration system, method, device and storage medium for an API gateway, wherein the system includes an API gateway, a distributed message service center cluster and a log processing module; the distributed message service center cluster is connected to the API gateway and the log processing module respectively; the API gateway is used to collect the log data of the gateway, serialize the log data in combination with the log structure definition file, and send the serialized log data to the distributed message service center cluster; the distributed message service center cluster is used to store the serialized log data; the log processing module is used to pull the serialized log data from the distributed message service center cluster, re-serialize the serialized log data in combination with the log structure definition file, and obtain log data with no defined structure that can be used by business applications. The system improves the performance of collecting log data and the convenience of log integration, and solves the problems of low performance in collecting log data and inconvenient integration in the prior art.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of the structure of a log integration system for an API gateway provided in Example 1 of the present invention;

[0020] Figure 2 A schematic diagram of the structure of a log integration system for an API gateway provided in the second embodiment of the present invention;

[0021] Figure 3 An application diagram of a log integration system for an API gateway provided by an embodiment of the present invention;

[0022] Figure 4 A flow chart of a log integration method for an API gateway provided in Example 3 of the present invention;

[0023] Figure 5 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention. It should be understood that the various steps described in the method implementation mode of the present invention can be performed in different orders and / or in parallel. In addition, the method implementation mode may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0025] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0026] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, any variations of the terms "including" and "having" are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0027] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0029] Example 1

[0030] Figure 1 This is a structural diagram of a log integration system for an API gateway provided in Example 1 of the present invention. The system is applicable to situations where data in an API gateway is collected and integrated. The system can execute a log integration method for an API gateway.

[0031] like Figure 1 As shown, a log integration system for an API gateway provided in the first embodiment of the present invention includes: an API gateway 100, a distributed message service center cluster 200, and a log processing module 300; the distributed message service center cluster 200 is connected to the API gateway 100 and the log processing module 300 respectively;

[0032] The API gateway 100 is used to collect the log data of the gateway, serialize the log data in combination with the log structure definition file, and send the serialized log data to the distributed message service center cluster 200;

[0033] Distributed message service center cluster 200, used to store the serialized log data;

[0034] The log processing module 300 is used to pull the serialized log data from the distributed message service center cluster 200, and re-serialize the serialized log data in combination with the log structure definition file to obtain log data with no defined structure that can be used by business applications.

[0035] Among them, the API gateway 100 can be a tool for API management and service governance that integrates functions such as configuration release, environment management, access authentication, user authentication, and access control. The distributed message service center cluster 200 can be a module for centrally managing the log data sent by the API gateway 100. The log processing module 300 can be a module for processing log data. Log data can be process event record data generated by the recording system. The type of log data and the data included are not limited in this embodiment. For example, the log data may include a request header, request source IP, request body, response header, response body, forwarding delay time, request time and other data. The log structure definition file can be a file that defines the data structure of the log data, and different types of log data correspond to different log structure definition files. The business application can be a software program or a group of programs that provide business capabilities, such as security analysis applications, situational awareness applications, and monitoring and alarm applications.

[0036] In this example, serialization is the process of converting an object's state into a format that can be persisted or transmitted, such as a byte stream, so that it can be written to a file, stored in a database, or sent over a network. Deserialization is the reverse process of serialization, which refers to reading data from a storage or transmission medium and reconstructing the object's original state.

[0037] This embodiment can collect the log data of the gateway through the API gateway 100, and efficiently serialize the log data in combination with the log structure definition file, and send the serialized log data to the distributed message service center cluster 200. The distributed message service center cluster 200 can store the serialized log data; the log processing module 300 can pull the serialized log data from the distributed message service center cluster 200, and re-serialize the serialized log data in combination with the log structure definition file to obtain log data with no defined structure, so that it can be used by business applications.

[0038] A log integration system for an API gateway provided in a first embodiment of the present invention includes an API gateway, a distributed message service center cluster, and a log processing module; the distributed message service center cluster is connected to the API gateway and the log processing module respectively; the API gateway is used to collect the log data of the gateway, serialize the log data in combination with the log structure definition file, and send the serialized log data to the distributed message service center cluster; the distributed message service center cluster is used to store the serialized log data; the log processing module is used to pull the serialized log data from the distributed message service center cluster, re-serialize the serialized log data in combination with the log structure definition file, and obtain log data with no defined structure that can be used by business applications. This system improves the performance of collecting log data and the convenience of log integration, and solves the problems of low performance in collecting log data and inconvenient integration in the prior art.

[0039] Based on the above embodiment, a modified embodiment of the above embodiment is proposed. It should be noted that, in order to simplify the description, only the differences from the above embodiment are described in the modified embodiment.

[0040] In one embodiment, the log processing module 300 is further configured to:

[0041] The unstructured log data is sent to the second message queue of the distributed message service center cluster 200 for use by business applications; wherein the second message queue is used to store the unstructured log data sent by the log processing module.

[0042] The second message queue may be a queue for storing deserialized data.

[0043] In this embodiment, the log processing module 300 may also send log data with no defined structure to the second message queue of the distributed message service center cluster 200, so that other business applications can obtain the log data from the second message queue when calling the log data.

[0044] Example 2

[0045] Figure 2 This is a schematic diagram of the structure of a log integration system for an API gateway provided by Example 2 of the present invention. This Example 2 is optimized based on the above-mentioned embodiments. For details not yet fully described in this Example, please refer to Example 1.

[0046] like Figure 2As shown, a log integration system for an API gateway is provided in the second embodiment of the present invention: the API gateway 100 includes an asynchronous log collection module 110, and the asynchronous log collection module includes a log serialization module 111, a batch compression module 112 and a batch sending module 113;

[0047] The log serialization module 111 is used to serialize the log data in combination with the log structure definition file and send the serialized log data to the batch compression module 112;

[0048] The batch compression module 112 is used to batch compress the serialized log data and send the compressed log data to the batch sending module 113;

[0049] The batch sending module 113 is used to send the compressed log data to the distributed message service center cluster 200.

[0050] The asynchronous log collection module 110 may be a module that collects and processes log data from the gateway. The log serialization module 111 may be a module that serializes log data. The batch compression module 112 may be a module that compresses log data. The batch sending module 113 may be a module that sends log data.

[0051] In this embodiment, the API gateway 100 includes an asynchronous log collection module 110, which can collect log data in the gateway. The asynchronous log collection module 110 can include a log serialization module 111, a batch compression module 112, and a batch sending module 113. Through the cooperation between the log serialization module 111, the batch compression module 112, and the batch sending module 113, the log data can be serialized, compressed, and sent.

[0052] A log integration system for an API gateway is provided in a second embodiment of the present invention. The API gateway includes an asynchronous log collection module, and the asynchronous log collection module includes a log serialization module, a batch compression module, and a batch sending module; the log serialization module is used to serialize the log data in combination with a log structure definition file, and send the serialized log data to the batch compression module; the batch compression module is used to batch compress each serialized log data, and send the compressed log data to the batch sending module; the batch sending module is used to send the compressed log data to the distributed message service center cluster. The system processes log data in parallel through the log serialization module, the batch compression module, and the batch sending module, which can improve the efficiency of integrating log data, and by compressing the log data, it can reduce the size of the transmitted message and reduce network resource consumption.

[0053] In one embodiment, the API gateway 100 also includes a server log network port.

[0054] Batch sending module 113, specifically used to send the compressed log data to the server log network port;

[0055] The server log network port is used to send the compressed log data to the distributed message service center cluster 200.

[0056] The server log network port may be a network port in the server.

[0057] In this embodiment, the batch sending module 113 can send the compressed log data to the server log network port, and the server log network port can send the compressed log data to the distributed message service center cluster 200. For example, Figure 3 An application diagram of a log integration system for an API gateway provided by an embodiment of the present invention is as follows Figure 3 As shown in the figure, the server log network port is a separate network port in the server network port and is not shared with the business network port. This ensures the stability of the business network port while also ensuring the performance of the log network port.

[0058] In one embodiment, the log serialization module 111 is specifically configured to:

[0059] Create a message object;

[0060] Assign the value of the fixed-type log field in the log data to the corresponding field of the message object;

[0061] Serialize the non-fixed type log fields in the log data, and assign the serialized values ​​to the corresponding fields of the message object;

[0062] The assigned message object is serialized through the log structure definition file and serialization tool to obtain the serialized log data.

[0063] The message object (map) can be a data structure used to store key-value pairs. A fixed type can be a type whose value changes infrequently, while a non-fixed type can be a type whose value changes frequently and dynamically. A log field can be a field in the log data. For example, fields like request headers and response headers in log data change dynamically, while fields like the request path and requester are fixed-type fields. The serialization tool can be a serialization tool based on a log structure definition file, such as protobuf.

[0064] In this embodiment, when performing serialization, the log serialization module 111 can first create a message object, assign the value of the fixed-type log field in the log data to the corresponding field of the message object, and serialize the non-fixed-type log field in the log data into JSON format to perform the first serialization, and assign the serialized value to the corresponding field of the message object, and then perform a second serialization through the log structure definition file and the serialization tool, serialize the assigned message object, and obtain the serialized log data. Exemplarily, first create a message object for storing message data, assign fixed-type log fields to the message object, such as the request source IP and request body fields; then serialize the non-fixed-type fields, such as performing Json serialization on the request header, and assign the serialized value to the fixed attribute in the current message object, use the serialization-defined structure file (.proto) and serialization tool (protobuf) to serialize the message object, and save it to the local memory array.

[0065] API gateway log data is generated by the API gateway. The API gateway parses protocol packets according to the protocol and assembles them into log messages. For example, protocol messages in memory are serialized into log message structured data. Log messages typically also need to be expanded with latency statistics or contextual business information. Currently, cloud-native serialization formats include JSON and Protobuf. These can be mainly divided into two categories: serialization tools that rely on log structured definition files, such as Protobuf, and serialization tools that do not rely on log structured definition files, such as JSON. Serialization tools that rely on log structured definition files offer high serialization performance and small serialization footprint, but they do not support serialization of data with variable structures and also rely on log structured definition files for deserialization, resulting in relatively high integration costs. Serialization tools that do not rely on log structured definition files support serialization of data with variable structures and do not rely on log structured definition files, making serialization and deserialization more convenient. However, their serialization performance is relatively low, and the serialized messages are relatively large (including field names and structure information). The performance of serialization tools that rely on structured definition files is more than 10 times higher than that of serialization tools that do not rely on log structured definition files, and the size of serialized messages is also about 1 / 3 less.

[0066] This embodiment combines the above analysis and selects a combined serialization method. During the first serialization, the non-fixed structure sub-log fields are serialized first, and then serialized by relying on the log structure definition file. This can meet the integration scenarios of various types of quota log data. In addition, in the scenario where all fields are fixed structured, the sequence is simplified to a single layer of serialization, which is equivalent to a single serialization relying on the log structure definition file, and the performance will be maximized.

[0067] In one embodiment, the batch compression module 112 is specifically configured to:

[0068] When the number of locally stored log data reaches a preset threshold, or a preset sending time is reached, each serialized log data is batch compressed and the compressed log data is sent to the batch sending module 113 .

[0069] The preset threshold and the preset sending time can be set according to actual conditions.

[0070] In this embodiment, the batch compression module 112 can batch compress each serialized log data when the number of locally stored log data reaches a preset threshold, or when the current time reaches a preset sending time, and send the compressed log data to the batch sending module 113.

[0071] In terms of log caching, this embodiment can cache logs in the memory of the gateway service node, and no longer use traditional hard disk storage, so the API gateway does not need to have high storage performance. However, since the memory of the API gateway server is also limited, the cache size and cache refresh time must be controlled. Therefore, when the number of cached log data reaches a preset threshold or reaches a specified preset sending time (or refresh time), this embodiment will asynchronously send the log data in batches to the distributed message service center cluster through the batch sending module. On the network, the smaller the transmitted message, the less network consumption. Because the message queue of the distributed message service center cluster needs to collect data from all gateway nodes, it is necessary to use a higher-performance distributed queue, such as Kafka.

[0072] In one embodiment, the log processing module 300 is specifically configured to:

[0073] Pull the serialized log data from the first message queue of the distributed message service center cluster 200; the first message queue is used to store the serialized log data received from the API gateway 100;

[0074] Deserialize the serialized log data using the log structure definition file and a deserialization tool to obtain a deserialized message object;

[0075] Deserialize the non-fixed type log field in the deserialized message object to obtain a secondary serialized message object;

[0076] The message object after secondary serialization is serialized in a serialization manner with no defined structure to obtain log data with no defined structure.

[0077] The first message queue can be a queue for storing serialized log data sent by the API gateway. The deserialization tool can be a tool corresponding to the tool used to serialize the log data, or can be the same as the serialization tool. For example, the deserialization tool can also be protobuf.

[0078] In this embodiment, the log processing module 300 can pull serialized log data from the first message queue of the distributed message service center cluster 200. Since the serialized log data is serialized through a two-layer serialization method, when deserializing, the serialized log data can be first deserialized in combination with the log structure definition file and the deserialization tool to obtain a deserialized message object, and then the non-fixed type log fields in the deserialized message object are deserialized to obtain a secondary serialized message object. Finally, the secondary serialized message object is serialized through a serialization method with no defined structure to obtain log data with no defined structure. For example, after obtaining the serialized log data, it is first decompressed, and then the message object is deserialized into a message object using the serialization-defined structure file (.proto) and the serialization tool (protobuf), and the non-fixed JSON fields are deserialized into MAP, and the standard JSON object is assembled and serialized to obtain log data with no defined structure.

[0079] In terms of the integration and use of log data, more attention is paid to the convenience of log integration. For the convenience of integration, this embodiment re-serializes the log data through the log processing module. The re-serialization process includes deserialization and re-serialization. Through re-serialization, the serialized log data can be in a format that does not depend on the log structure definition file, such as JSON format, so that the integrated application can be easily integrated.

[0080] The API gateway log integration system of this embodiment is not limited by the performance limitations of server storage disks, maximizing network bandwidth and computing resource savings while also allowing business applications to easily integrate API gateway traffic logs. This allows for stable collection and integration of gateway logs in high-concurrency scenarios.

[0081] Example 3

[0082] Figure 4 This is a flow chart of a log integration method for an API gateway provided in Example 3 of the present invention. This method is applicable to situations where data in an API gateway is collected and integrated, and this method can be executed by a log integration system of an API gateway.

[0083] like Figure 4As shown, a log integration method for an API gateway provided in the third embodiment of the present invention includes the following steps:

[0084] S310. Collect the log data of the gateway through the API gateway, serialize the log data in combination with the log structure definition file, and send the serialized log data to the distributed message service center cluster.

[0085] Among them, log data can be process event record data generated by the recording system, and the type of log data and the data included in this embodiment are not limited to this. For example, the log data may include a request header, a request source IP, a request body, a response header, a response body, a forwarding delay time, a request time, and other data. The log structure definition file can be a file that defines the data structure of the log data, and the log structure definition files corresponding to different types of log data are different.

[0086] In this embodiment, the log data of the gateway can be collected through the API gateway, the log data can be serialized in combination with the log structure definition file, and the serialized log data can be sent to the distributed message service center cluster.

[0087] S320. Pull the serialized log data from the distributed message service center cluster through the log processing module, and re-serialize the serialized log data in combination with the log structure definition file to obtain log data with no defined structure that can be used by business applications.

[0088] Among them, the business application can be a software program or a group of programs that provide business capabilities, such as security analysis applications, situational awareness applications, and monitoring and alarm applications.

[0089] In this embodiment, the serialized log data can be pulled from the distributed message service center cluster through the log processing module, and the serialized log data can be re-serialized in combination with the log structure definition file to obtain log data with no defined structure that can be used by business applications.

[0090] This embodiment provides a log integration method for an API gateway, comprising: collecting gateway log data through the API gateway, serializing the log data in conjunction with a log structure definition file, and sending the serialized log data to a distributed message service center cluster; pulling the serialized log data from the distributed message service center cluster through a log processing module, reserializing the serialized log data in conjunction with the log structure definition file, and obtaining unstructured log data that can be used by business applications. This method improves the performance of log data collection and the convenience of log integration, solving the problems of low log data collection performance and inconvenient integration in the prior art.

[0091] Furthermore, the API gateway includes an asynchronous log collection module. Accordingly, the API gateway collects the log data of the gateway, serializes the log data in combination with the log structure definition file, and sends the serialized log data to the distributed message service center cluster, including:

[0092] The log data is serialized by the log serialization module in the asynchronous log collection module in combination with the log structure definition file, and the serialized log data is sent to the batch compression module in the asynchronous log collection module;

[0093] The batch compression module batch compresses the serialized log data and sends the compressed log data to the batch sending module in the asynchronous log collection module;

[0094] The compressed log data is sent to the distributed message service center cluster through a batch sending module.

[0095] Furthermore, the sending of the compressed log data to the distributed message service center cluster through a batch sending module includes:

[0096] Send the compressed log data to the server log network port through the batch sending module;

[0097] The compressed log data is sent to the distributed message service center cluster through the server log network port.

[0098] Furthermore, the log data is serialized by the log serialization module in the asynchronous log collection module in combination with the log structure definition file, including:

[0099] Create a message object;

[0100] Assign the value of the fixed-type log field in the log data to the corresponding field of the message object;

[0101] Serialize the non-fixed type log fields in the log data, and assign the serialized values ​​to the corresponding fields of the message object;

[0102] The assigned message object is serialized through the log structure definition file and serialization tool to obtain the serialized log data.

[0103] Furthermore, the batch compression module batch compresses each serialized log data and sends the compressed log data to the batch sending module in the asynchronous log collection module, including:

[0104] When the number of locally stored log data reaches a preset threshold, or the preset sending time is reached, each serialized log data is batch compressed and sent to the batch sending module in the asynchronous log collection module.

[0105] Furthermore, the log processing module pulls the serialized log data from the distributed message service center cluster, and re-serializes the serialized log data in combination with the log structure definition file to obtain log data with no defined structure that can be used by business applications, including:

[0106] Pulling the serialized log data from the first message queue of the distributed message service center cluster; the first message queue is used to store the serialized log data received from the API gateway;

[0107] Deserialize the serialized log data using the log structure definition file and a deserialization tool to obtain a deserialized message object;

[0108] Deserialize the non-fixed type log field in the deserialized message object to obtain a secondary serialized message object;

[0109] The message object after secondary serialization is serialized in a serialization manner with no defined structure to obtain log data with no defined structure.

[0110] Furthermore, the method further comprises:

[0111] Sending the unstructured log data to the second message queue of the distributed message service center cluster for use by business applications;

[0112] The second message queue is used to store log data with no defined structure sent by the log processing module.

[0113] Example 4

[0114] Figure 5 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0115] like Figure 5 As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0116] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0117] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, digital signal processors (DSPs), and any other suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the log integration method of the API gateway.

[0118] In some embodiments, the log integration method of the API gateway can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log integration method of the API gateway described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the log integration method of the API gateway by any other appropriate means (for example, by means of firmware).

[0119] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0124] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0125] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0126] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A log integration system for an API gateway, characterized in that: The system includes an API gateway, a distributed message service center cluster and a log processing module; the distributed message service center cluster is connected to the API gateway and the log processing module respectively; The API gateway is used to collect the log data of the gateway, serialize the log data in combination with the log structure definition file, and send the serialized log data to the distributed message service center cluster; The distributed message service center cluster is used to store the serialized log data; The log processing module is used to pull the serialized log data from the distributed message service center cluster, and re-serialize the serialized log data in combination with the log structure definition file to obtain log data with no defined structure that can be used by business applications.

2. The system according to claim 1, wherein: The API gateway includes an asynchronous log collection module, which includes a log serialization module, a batch compression module and a batch sending module; The log serialization module is used to serialize the log data in combination with the log structure definition file, and send the serialized log data to the batch compression module; The batch compression module is used to batch compress the serialized log data and send the compressed log data to the batch sending module; The batch sending module is used to send the compressed log data to the distributed message service center cluster.

3. The system according to claim 2, characterized in that The API gateway also includes a server log port. The batch sending module is specifically used to send the compressed log data to the server log network port; The server log network port is used to send the compressed log data to the distributed message service center cluster.

4. The system according to claim 2, wherein: The log serialization module is specifically used for: Create a message object; Assign the value of the fixed-type log field in the log data to the corresponding field of the message object; Serialize the non-fixed type log fields in the log data, and assign the serialized values ​​to the corresponding fields of the message object; The assigned message object is serialized through the log structure definition file and serialization tool to obtain the serialized log data.

5. The system according to claim 2, wherein: The batch compression module is specifically used for: When the number of locally stored log data reaches a preset threshold, or reaches a preset sending time, each serialized log data is batch compressed and the compressed log data is sent to the batch sending module.

6. The system according to claim 1, wherein: The log processing module is specifically used for: Pulling the serialized log data from the first message queue of the distributed message service center cluster; the first message queue is used to store the serialized log data received from the API gateway; Deserialize the serialized log data using the log structure definition file and a deserialization tool to obtain a deserialized message object; Deserialize the non-fixed type log field in the deserialized message object to obtain a secondary serialized message object; The message object after secondary serialization is serialized in a serialization manner with no defined structure to obtain log data with no defined structure.

7. The system according to claim 1, wherein: The log processing module is also used to: Sending the unstructured log data to the second message queue of the distributed message service center cluster for use by business applications; The second message queue is used to store log data with no defined structure sent by the log processing module.

8. A log integration method for an API gateway, characterized in that: The method comprises: Collect the gateway's log data through the API gateway, serialize the log data in combination with the log structure definition file, and send the serialized log data to the distributed message service center cluster; The serialized log data is pulled from the distributed message service center cluster through a log processing module, and the serialized log data is reserialized in combination with the log structure definition file to obtain log data with no defined structure that can be used by business applications.

9. An electronic device, characterized in that: The device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log integration method of the API gateway described in claim 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a processor to implement the log integration method of the API gateway according to claim 8 when executed.

Images