A log integration system, method, device, and storage medium for an API gateway.

Abstract

This invention discloses a log integration system, method, device, and storage medium for an API gateway. The system includes an API gateway, a distributed message service center cluster, and a log processing module. The distributed message service center cluster is connected to both the API gateway and the log processing module. The API gateway collects log data from the gateway, serializes the log data using a log structure definition file, and sends the serialized log data to the distributed message service center cluster. The distributed message service center cluster stores the serialized log data. The log processing module retrieves the serialized log data from the distributed message service center cluster, re-serializes the serialized log data using the log structure definition file, and obtains unstructured log data usable by business applications. This system improves the performance of log data collection and the convenience of log integration.

Description

Technical Field

[0001] This invention relates to the field of log processing technology, and in particular to a log integration system, method, device, and storage medium for an API gateway. Background Technology

[0002] With the advancement of digital transformation, Application Programming Interface (API) gateways play a crucial role in enterprise digital transformation, helping enterprises achieve digital innovation, business optimization, and improved customer experience. By effectively utilizing the functions and advantages of API gateways, enterprises can better adapt to market changes, improve operational efficiency, and achieve sustainable growth and competitive advantage. However, the explosive growth in the number of APIs and concurrent access volume has brought unprecedented performance pressure to API gateway application scenarios, such as API gateway log collection and integration.

[0003] In traditional enterprises, API gateways are typically deployed across multiple clusters and nodes. For log collection, they need to centrally collect proxy traffic logs from all nodes. These logs need to be used by various business applications, such as security analytics, situational awareness, and monitoring / alerting. However, with existing methods, other business applications cannot easily integrate the API gateway's traffic logs. Summary of the Invention

[0004] This invention provides a log integration system, method, device, and storage medium for an API gateway to solve the problems of low log data collection performance and inconvenient integration in the prior art.

[0005] According to one aspect of the present invention, a log integration system for an API gateway is provided, the system comprising an API gateway, a distributed message service center cluster, and a log processing module; the distributed message service center cluster is connected to the API gateway and the log processing module respectively;

[0006] The API gateway is used to collect the gateway's log data, serialize the log data in conjunction with the log structure definition file, and send the serialized log data to the distributed message service center cluster.

[0007] The distributed message service center cluster is used to store the serialized log data;

[0008] The log processing module is used to pull the serialized log data from the distributed message service center cluster, and re-serialize the serialized log data in conjunction with the log structure definition file to obtain undefined log data that can be used by business applications.

[0009] According to another aspect of the present invention, a log integration method for an API gateway is provided, the method comprising:

[0010] Log data from the API gateway is collected, and the log data is serialized using the log structure definition file. The serialized log data is then sent to the distributed message service center cluster.

[0011] The log processing module pulls the serialized log data from the distributed message service center cluster, and re-serializes the serialized log data using the log structure definition file to obtain undefined log data that can be used by business applications.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and

[0013] A memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the log integration method of the API gateway according to any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the log integration method of the API gateway according to any embodiment of the present invention.

[0016] This invention discloses a log integration system, method, device, and storage medium for an API gateway. The system includes an API gateway, a distributed message service center cluster, and a log processing module. The distributed message service center cluster is connected to both the API gateway and the log processing module. The API gateway collects log data from the gateway, serializes the log data using a log structure definition file, and sends the serialized log data to the distributed message service center cluster. The distributed message service center cluster stores the serialized log data. The log processing module retrieves the serialized log data from the distributed message service center cluster, re-serializes the serialized log data using the log structure definition file, and obtains undefined log data usable by business applications. This system improves the performance of log data collection and the convenience of log integration, solving the problems of low collection performance and inconvenient integration in existing technologies.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of an API gateway log integration system provided in Embodiment 1 of the present invention;

[0020] Figure 2 This is a schematic diagram of the structure of an API gateway log integration system provided in Embodiment 2 of the present invention;

[0021] Figure 3 This is an application diagram of an API gateway log integration system provided in an embodiment of the present invention;

[0022] Figure 4 This is a flowchart illustrating an API gateway log integration method provided in Embodiment 3 of the present invention;

[0023] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. It should be understood that the various steps described in the method embodiments of the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0025] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, any variations of the terms "comprising" and "having," etc., are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0028] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0029] Example 1

[0030] Figure 1 This is a schematic diagram of the structure of an API gateway log integration system provided in Embodiment 1 of the present invention. The system is applicable to the collection and integration of data in an API gateway and can execute an API gateway log integration method.

[0031] like Figure 1 As shown, an API gateway log integration system provided in Embodiment 1 of the present invention includes: an API gateway 100, a distributed message service center cluster 200, and a log processing module 300; the distributed message service center cluster 200 is connected to the API gateway 100 and the log processing module 300 respectively.

[0032] API Gateway 100 is used to collect log data from the gateway, serialize the log data in conjunction with the log structure definition file, and send the serialized log data to the distributed message service center cluster 200.

[0033] Distributed message service center cluster 200 is used to store the serialized log data;

[0034] The log processing module 300 is used to pull the serialized log data from the distributed message service center cluster 200, and re-serialize the serialized log data in conjunction with the log structure definition file to obtain undefined log data that can be used by business applications.

[0035] API Gateway 100 can be a tool integrating API management and service governance functions such as configuration publishing, environment management, access authentication, user authentication, and access control. Distributed Message Service Center Cluster 200 can be a module for centrally managing the log data sent by API Gateway 100. Log Processing Module 300 can be a module for processing log data. Log data can be record data of procedural events generated by the system; the type and included data of log data are not limited in this embodiment. For example, log data may include request headers, request source IP, request body, response headers, response body, forwarding delay time, request time, and other data. Log structure definition file can be a file that defines the data structure of log data; different types of log data correspond to different log structure definition files. Business applications can be software programs or a set of programs that provide business capabilities, such as security analysis applications, situational awareness applications, and monitoring and alarm applications.

[0036] In this embodiment, serialization is the process of converting an object's state into a format that can be persisted or transmitted, such as a byte stream, so that it can be written to a file, stored in a database, or sent over a network. Deserialization is the reverse process of serialization, referring to reading data from a storage or transmission medium and reconstructing the object's original state.

[0037] In this embodiment, the API gateway 100 can collect log data from the gateway and efficiently serialize the log data using a log structure definition file. The serialized log data is then sent to the distributed message service center cluster 200, which can store the serialized log data. The log processing module 300 can then pull the serialized log data from the distributed message service center cluster 200 and re-serialize it using the log structure definition file to obtain unstructured log data, which can then be used by business applications.

[0038] This invention provides a log integration system for an API gateway, comprising an API gateway, a distributed message service center cluster, and a log processing module. The distributed message service center cluster is connected to both the API gateway and the log processing module. The API gateway collects log data from the gateway, serializes the log data using a log structure definition file, and sends the serialized log data to the distributed message service center cluster. The distributed message service center cluster stores the serialized log data. The log processing module retrieves the serialized log data from the distributed message service center cluster, re-serializes the serialized log data using the log structure definition file, and obtains unstructured log data usable by business applications. This system improves the performance of log data collection and the convenience of log integration, solving the problems of low collection performance and inconvenient integration in existing technologies.

[0039] Based on the above embodiments, modified embodiments of the above embodiments are proposed. It should be noted that, in order to keep the description brief, only the differences from the above embodiments are described in the modified embodiments.

[0040] In one embodiment, the log processing module 300 is further configured to:

[0041] The unstructured log data is sent to the second message queue of the distributed message service center cluster 200 for use by business applications; wherein, the second message queue is used to store the unstructured log data sent by the log processing module.

[0042] The second message queue can be a queue that stores data that has already been deserialized.

[0043] In this embodiment, the log processing module 300 can also send unstructured log data to the second message queue of the distributed message service center cluster 200, so that other business applications can obtain the log data from the second message queue when calling the log data.

[0044] Example 2

[0045] Figure 2 This is a schematic diagram of the structure of an API gateway log integration system provided in Embodiment 2 of the present invention. Embodiment 2 is an optimization based on the above embodiments. For details not covered in this embodiment, please refer to Embodiment 1.

[0046] like Figure 2As shown in Embodiment 2 of the present invention, an API gateway log integration system is provided: API gateway 100 includes an asynchronous log collection module 110, which includes a log serialization module 111, a batch compression module 112, and a batch sending module 113;

[0047] The log serialization module 111 is used to serialize the log data in conjunction with the log structure definition file and send the serialized log data to the batch compression module 112.

[0048] The batch compression module 112 is used to perform batch compression on each serialized log data and send the compressed log data to the batch sending module 113.

[0049] The batch sending module 113 is used to send the compressed log data to the distributed message service center cluster 200.

[0050] The asynchronous log acquisition module 110 is used to collect and process log data from the gateway. The log serialization module 111 is used to serialize log data. The batch compression module 112 is used to compress log data. The batch sending module 113 is used to send log data.

[0051] In this embodiment, the API gateway 100 includes an asynchronous log collection module 110, which can collect log data from the gateway. The asynchronous log collection module 110 may include a log serialization module 111, a batch compression module 112, and a batch sending module 113. Through the cooperation between the log serialization module 111, the batch compression module 112, and the batch sending module 113, log data can be serialized, compressed, and sent.

[0052] This invention provides a log integration system for an API gateway, as shown in Embodiment 2. The API gateway includes an asynchronous log collection module, which further comprises a log serialization module, a batch compression module, and a batch sending module. The log serialization module serializes the log data using a log structure definition file and sends the serialized log data to the batch compression module. The batch compression module performs batch compression on each serialized log data and sends the compressed log data to the batch sending module. The batch sending module sends the compressed log data to the distributed message service center cluster. This system processes log data in parallel through the log serialization, batch compression, and batch sending modules, improving the efficiency of log data integration. Furthermore, by compressing the log data, the size of the transmitted messages is reduced, thus lowering network resource consumption.

[0053] In one embodiment, the API gateway 100 also includes a server log network port, and correspondingly...

[0054] The batch sending module 113 is specifically used to send the compressed log data to the server log network port;

[0055] The server log port is used to send the compressed log data to the distributed message service center cluster 200.

[0056] The server log port can be a network port on the server.

[0057] In this embodiment, the batch sending module 113 can send the compressed log data to the server log network port, and the server log network port can send the compressed log data to the distributed message service center cluster 200. For example, Figure 3 This is an application diagram of an API gateway log integration system provided in an embodiment of the present invention, such as... Figure 3 As shown, the server log port is a separate port in the server network interface and is not shared with the business network interface. This ensures the stability of the business network interface while also guaranteeing the performance of the log port.

[0058] In one embodiment, the log serialization module 111 is specifically used for:

[0059] Create a message object;

[0060] Assign the values ​​of the fixed-type log fields in the log data to the corresponding fields of the message object;

[0061] Serialize the non-fixed type log fields in the log data, and assign the serialized values ​​to the fields corresponding to the message object;

[0062] The assigned message object is serialized using a log structure definition file and a serialization tool to obtain serialized log data.

[0063] In this context, a message object (map) can be a data structure used to store key-value pairs. Fixed types can be types of values ​​that don't change frequently, while non-fixed types can be types of values ​​that change frequently and dynamically. Log fields can be fields from log data. For example, fields like request headers and response headers in log data change dynamically, while fields like request path and requester are fixed-type fields. Serialization tools can be serialization tools based on the log structure definition file; for example, a serialization tool like protobuf.

[0064] In this embodiment, when performing serialization, the log serialization module 111 can first create a message object, assign the values ​​of fixed-type log fields from the log data to the corresponding fields of the message object, and serialize the non-fixed-type log fields from the log data into JSON format for the first serialization. The serialized values ​​are then assigned to the corresponding fields of the message object. A second serialization is then performed using a log structure definition file and a serialization tool to serialize the assigned message object, resulting in serialized log data. For example, a message object is first created to store the message data, and fixed-type log fields, such as the request source IP and request body, are assigned to this message object. Then, non-fixed-type fields are serialized, such as request headers, using JSON serialization. The serialized values ​​are assigned to fixed attributes in the current message object. The message object is then serialized using a predefined serialization structure file (.proto) and a serialization tool (protobuf), and saved to a local memory array.

[0065] API gateway log data is generated by the API gateway, which parses protocol data packets according to the protocol and assembles them into log messages. For example, it serializes protocol messages in memory into log message structure data. Log messages typically also need to be extended with latency statistics or contextual business information. Currently, cloud-native serialization formats include JSON and Protobuf, which can be mainly divided into two categories: serialization tools based on log structure definition files, such as Protobuf, and serialization tools that do not rely on log structure definition files, such as JSON. Serialization tools that rely on log structure definition files have high serialization performance and occupy less space after serialization, but they do not support the serialization of data with variable structures, and deserialization also depends on the log structure definition file, resulting in relatively high integration costs. Serialization tools that do not rely on log structure definition files support the serialization of data with variable structures, do not require a log structure definition file, and are very convenient for serialization and deserialization. However, their serialization performance is relatively low, and the serialized message is relatively large (including field names and structure information). Serialization tools that rely on file structure definitions are more than 10 times faster than those that do not, and the serialized message size is also about one-third smaller.

[0066] This embodiment, based on the above analysis, selects a combined serialization method. During the first serialization, non-fixed-structure word log fields are serialized first, followed by serialization via a log structure definition file. This satisfies the integration scenarios of various types of quota log data. Furthermore, in scenarios with all fixed-structure fields, the serialization is simplified to a single layer, equivalent to a single serialization via the log structure definition file, thus maximizing performance.

[0067] In one embodiment, the batch compression module 112 is specifically used for:

[0068] When the number of log data entries stored locally reaches a preset threshold or a preset sending time is reached, the serialized log data is compressed in batches and the compressed log data is sent to the batch sending module 113.

[0069] The preset threshold and preset sending time can be set according to the actual situation.

[0070] In this embodiment, the batch compression module 112 can perform batch compression on each serialized log data when the number of log data stored locally reaches a preset threshold, or when the current time reaches a preset sending time, and then send the compressed log data to the batch sending module 113.

[0071] In this embodiment, logs can be cached in the memory of the gateway service node, eliminating the need for traditional hard disk storage and thus reducing the requirement for high storage performance from the API gateway. However, since the API gateway server's memory is also limited, the cache size and refresh time must be controlled. Therefore, this embodiment asynchronously sends the log data to the distributed message service center cluster in batches via the batch sending module when the number of cached log data reaches a preset threshold or a specified preset sending time (or refresh time). On the network, smaller transmitted packets consume less network resources. Because the message queue of the distributed message service center cluster needs to collect data from all gateway nodes, a high-performance distributed queue, such as Kafka, is required.

[0072] In one embodiment, the log processing module 300 is specifically used for:

[0073] The serialized log data is pulled from the first message queue of the distributed message service center cluster 200; the first message queue is used to store the serialized log data received from the API gateway 100.

[0074] By combining the log structure definition file and the deserialization tool, the serialized log data is deserialized to obtain a deserialized message object;

[0075] The non-fixed type log fields in the deserialized message object are deserialized to obtain a secondary serialized message object;

[0076] The message object after secondary serialization is serialized using an undefined serialization method to obtain log data without a defined structure.

[0077] The first message queue can be a queue that stores the serialized log data sent by the API gateway. The deserialization tool can be the same tool used to serialize the log data, or it can be the same as the serialization tool. For example, the deserialization tool could also be protobuf.

[0078] In this embodiment, the log processing module 300 can pull serialized log data from the first message queue of the distributed message service center cluster 200. Since the serialized log data is serialized through a two-layer serialization process, during deserialization, the serialized log data can first be deserialized using a log structure definition file and a deserialization tool to obtain a deserialized message object. Then, the non-fixed-type log fields in the deserialized message object are deserialized to obtain a secondary serialized message object. Finally, the secondary serialized message object is serialized using a non-structured serialization method to obtain unstructured log data. For example, after obtaining the serialized log data, it is first decompressed, and then the predefined serialization structure file (.proto) and serialization tool (protobuf) are used to deserialize the message object into a message object. Non-fixed JSON fields are deserialized into a MAP, a standard JSON object is assembled and serialized to obtain unstructured log data.

[0079] In terms of log data integration and use, the convenience of log integration is of paramount importance. To facilitate integration, this embodiment uses a log processing module to re-serialize log data. The re-serialization process includes deserialization and re-serialization. Re-serialization enables the serialized log data to be in a format that does not depend on the log structure definition file, such as JSON format, thus making it easy for integrated applications to integrate.

[0080] This embodiment presents a log integration system for an API gateway that is not limited by the performance constraints of server storage disks, thus maximizing the conservation of network bandwidth and computing resources. It also allows business applications to easily integrate API gateway traffic logs. In high-concurrency scenarios involving the API gateway, it ensures stable collection and integration of gateway logs.

[0081] Example 3

[0082] Figure 4 This is a flowchart illustrating an API gateway log integration method according to Embodiment 3 of the present invention. This method is applicable to the collection and integration of data in an API gateway and can be executed by the API gateway's log integration system.

[0083] like Figure 4As shown in Embodiment 3 of the present invention, an API gateway log integration method includes the following steps:

[0084] S310. Collect log data from the API gateway, serialize the log data using the log structure definition file, and send the serialized log data to the distributed message service center cluster.

[0085] Log data can be records of procedural events generated by the system. This embodiment does not limit the type or included data of the log data. For example, log data may include request headers, request source IP, request body, response headers, response body, forwarding delay time, request time, and other data. The log structure definition file is a file that defines the data structure of the log data; different types of log data correspond to different log structure definition files.

[0086] In this embodiment, log data from the gateway can be collected through the API gateway, serialized using the log structure definition file, and then sent to the distributed message service center cluster.

[0087] S320. The log processing module pulls the serialized log data from the distributed message service center cluster, and re-serializes the serialized log data in conjunction with the log structure definition file to obtain undefined log data that can be used by business applications.

[0088] Among them, business applications can be software programs or a set of programs that provide business capabilities, such as security analysis applications, situational awareness applications, and monitoring and alarm applications.

[0089] In this embodiment, the serialized log data can be pulled from the distributed message service center cluster by the log processing module, and the serialized log data can be re-serialized in combination with the log structure definition file to obtain undefined log data that can be used by business applications.

[0090] This embodiment provides a log integration method for an API gateway, comprising: collecting log data from the API gateway, serializing the log data using a log structure definition file, and sending the serialized log data to a distributed message service center cluster; retrieving the serialized log data from the distributed message service center cluster using a log processing module, and re-serializing the serialized log data using the log structure definition file to obtain undefined log data usable by business applications. This method improves the performance of log data collection and the convenience of log integration, solving the problems of low log data collection performance and inconvenient integration in existing technologies.

[0091] Furthermore, the API gateway includes an asynchronous log collection module. Correspondingly, the process of collecting log data from the API gateway, serializing the log data using a log structure definition file, and sending the serialized log data to the distributed message service center cluster includes:

[0092] The log data is serialized by the log serialization module in the asynchronous log acquisition module in conjunction with the log structure definition file, and the serialized log data is sent to the batch compression module in the asynchronous log acquisition module.

[0093] The batch compression module compresses each serialized log data in batches, and then sends the compressed log data to the batch sending module in the asynchronous log collection module.

[0094] The compressed log data is sent to the distributed message service center cluster via the batch sending module.

[0095] Furthermore, the step of sending the compressed log data to the distributed message service center cluster via the batch sending module includes:

[0096] The compressed log data is sent to the server log port via the batch sending module;

[0097] The compressed log data is sent to the distributed message service center cluster via the server log network port.

[0098] Furthermore, the serialization of the log data through the log serialization module in the asynchronous log acquisition module in conjunction with the log structure definition file includes:

[0099] Create a message object;

[0100] Assign the values ​​of the fixed-type log fields in the log data to the corresponding fields of the message object;

[0101] Serialize the non-fixed type log fields in the log data, and assign the serialized values ​​to the fields corresponding to the message object;

[0102] The assigned message object is serialized using a log structure definition file and a serialization tool to obtain serialized log data.

[0103] Furthermore, the step of batch compressing the serialized log data using the batch compression module and sending the compressed log data to the batch sending module in the asynchronous log acquisition module includes:

[0104] When the number of log data entries stored locally reaches a preset threshold, or when a preset sending time is reached, the serialized log data is compressed in batches, and the compressed log data is sent to the batch sending module in the asynchronous log collection module.

[0105] Furthermore, the step of pulling the serialized log data from the distributed message service center cluster through the log processing module, and reserializing the serialized log data in conjunction with the log structure definition file to obtain undefined log data that can be used by business applications includes:

[0106] The serialized log data is retrieved from the first message queue of the distributed message service center cluster; the first message queue is used to store the serialized log data received from the API gateway.

[0107] By combining the log structure definition file and the deserialization tool, the serialized log data is deserialized to obtain a deserialized message object;

[0108] The non-fixed type log fields in the deserialized message object are deserialized to obtain a secondary serialized message object;

[0109] The message object after secondary serialization is serialized using an undefined serialization method to obtain log data without a defined structure.

[0110] Furthermore, the method also includes:

[0111] The undefined log data is sent to the second message queue of the distributed message service center cluster for use by business applications.

[0112] The second message queue is used to store unstructured log data sent by the log processing module.

[0113] Example 4

[0114] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0115] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0116] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0117] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the log integration method of an API gateway.

[0118] In some embodiments, the API gateway log integration method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the API gateway log integration method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the API gateway log integration method by any other suitable means (e.g., by means of firmware).

[0119] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0123] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0124] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0125] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0126] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A log integration system for an API gateway, characterized in that, The system includes an API gateway, a distributed message service center cluster, and a log processing module; the distributed message service center cluster is connected to the API gateway and the log processing module respectively. The API gateway is used to collect the gateway's log data, serialize the log data in conjunction with the log structure definition file, and send the serialized log data to the distributed message service center cluster. The distributed message service center cluster is used to store the serialized log data; The log processing module is used to pull the serialized log data from the distributed message service center cluster, and re-serialize the serialized log data in conjunction with the log structure definition file to obtain undefined log data that can be used by business applications. The API gateway includes an asynchronous log collection module, which in turn includes a log serialization module. The log serialization module is specifically used for: Create a message object; Assign the values ​​of the fixed-type log fields in the log data to the corresponding fields of the message object; Serialize the non-fixed type log fields in the log data into JSON format, and assign the serialized values ​​to the corresponding fields of the message object; The assigned message object is serialized using a log structure definition file and a serialization tool to obtain serialized log data; the serialization tool is protobuf.

2. The system according to claim 1, characterized in that, The asynchronous log collection module includes a batch compression module and a batch sending module; The log serialization module is used to serialize the log data in conjunction with the log structure definition file, and send the serialized log data to the batch compression module. The batch compression module is used to perform batch compression on each serialized log data and send the compressed log data to the batch sending module. The batch sending module is used to send the compressed log data to the distributed message service center cluster.

3. The system according to claim 2, characterized in that, The API gateway also includes a server log network port, and correspondingly... The batch sending module is specifically used to send the compressed log data to the server log network port; The server log port is used to send the compressed log data to the distributed message service center cluster.

4. The system according to claim 2, characterized in that, The batch compression module is specifically used for: When the number of log data entries stored locally reaches a preset threshold, or when a preset sending time is reached, the serialized log data is compressed in batches, and the compressed log data is sent to the batch sending module.

5. The system according to claim 1, characterized in that, The log processing module is specifically used for: The serialized log data is retrieved from the first message queue of the distributed message service center cluster; the first message queue is used to store the serialized log data received from the API gateway. By combining the log structure definition file and the deserialization tool, the serialized log data is deserialized to obtain a deserialized message object; The non-fixed type log fields in the deserialized message object are deserialized to obtain a secondary serialized message object; The message object after secondary serialization is serialized using an undefined serialization method to obtain log data without a defined structure.

6. The system according to claim 1, characterized in that, The log processing module is also used for: The undefined log data is sent to the second message queue of the distributed message service center cluster for use by business applications. The second message queue is used to store unstructured log data sent by the log processing module.

7. A method for integrating logs into an API gateway, characterized in that, The method includes: Log data from the API gateway is collected, and the log data is serialized using the log structure definition file. The serialized log data is then sent to the distributed message service center cluster. The log processing module pulls the serialized log data from the distributed message service center cluster, and re-serializes the serialized log data in conjunction with the log structure definition file to obtain undefined log data that can be used by business applications. The API gateway includes an asynchronous log collection module. The log data is serialized using a log serialization module within the asynchronous log collection module, in conjunction with a log structure definition file. This includes: Create a message object; Assign the values ​​of the fixed-type log fields in the log data to the corresponding fields of the message object; Serialize the non-fixed type log fields in the log data into JSON format, and assign the serialized values ​​to the corresponding fields of the message object; The assigned message object is serialized using a log structure definition file and a serialization tool to obtain serialized log data; the serialization tool is protobuf.

8. An electronic device, characterized in that, The device includes: at least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log integration method of the API gateway as described in claim 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the log integration method of the API gateway as described in claim 7.

Images