A log data stream generation method and device, and a storage medium

Abstract

The application discloses a log data stream generation method and device and a storage medium. The method comprises the following steps: determining configuration information of a target format of a log; the configuration information comprises at least one structure node, and each structure node comprises at least one entry node; traversing the entry nodes in each structure node to obtain an entry node set corresponding to each structure node; converting the entry node set corresponding to each structure node into a member variable set of a target programming language corresponding to each structure node; calling the member variable set of the target programming language corresponding to each structure node to generate a log object; and converting the log object into a log data stream. According to the technical scheme, manual intervention in the conversion process of a log definition format file is reduced, and the accuracy of log format conversion is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a log data stream generation method and device and storage medium. BACKGROUND

[0002] In recent years, with the continuous expansion of the game industry, the number of game players, online time and the amount of data generated by games are growing. In order to facilitate game background monitoring of key data in games and extraction of data for user analysis, most game servers need a complete and reliable tool chain to extract, collect and report key data in games.

[0003] With the continuous updating of game content and frequent iteration of game versions, the definition and content of these key logs may change frequently, so a simple and easy-to-read game log definition format is needed to facilitate editing of key log definitions, and high automation is needed to reduce human intervention in log collection and reporting storage links and reduce errors caused by human intervention.

[0004] In the mainstream technology, JSON (JavaScript Object Notation, a lightweight data exchange format), Protobuf (a platform-independent, language-independent, extensible and lightweight serialization data structure protocol), Thrift (an interface description language and binary communication protocol) and other data protocols are generally used to define game key logs. However, using data protocols such as JSON and Protobuf has high performance overhead, and the format complexity is high for non-professional developers. The solution of TDR is more efficient than using general data protocols for log definition, but TDR does not support Golang, and the functions of TDR itself are relatively complex and depend on the overall game server base framework solution, so the portability is poor.

[0005] Therefore, it is necessary to provide a log data stream generation method, device and storage medium which can reduce human intervention in the conversion process of log definition format files and improve the accuracy of log format conversion. SUMMARY

[0006] The present application provides a log data stream generation method, device and storage medium which can reduce human intervention in the conversion process of log definition format files and improve the accuracy of log format conversion.

[0007] In one aspect, the present application provides a log data stream generation method, which comprises:

[0008] configuration information of a target format of a log; the configuration information comprises at least one structure node, and each structure node comprises at least one entry node;

[0009] traversing the entry nodes in each structure node to obtain an entry node set corresponding to each structure node;

[0010] converting the entry node set corresponding to each structure node into a member variable set of a target programming language corresponding to each structure node;

[0011] calling the member variable set of the target programming language corresponding to each structure node to generate a log object;

[0012] converting the log object into a log data stream.

[0013] Another aspect provides a log data stream generation apparatus, and the apparatus comprises:

[0014] a configuration information determination module configured to determine configuration information of a target format of a log; the configuration information comprises at least one structure node, and each structure node comprises at least one entry node;

[0015] an entry node set determination module configured to traverse the entry nodes in each structure node to obtain an entry node set corresponding to each structure node;

[0016] a member variable set determination module configured to convert the entry node set corresponding to each structure node into a member variable set of a target programming language corresponding to each structure node;

[0017] a log object generation module configured to call the member variable set of the target programming language corresponding to each structure node to generate a log object;

[0018] a log data stream generation module configured to convert the log object into a log data stream.

[0019] Another aspect provides a computer storage medium, and the computer storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the log data stream generation method.

[0020] The log data stream generation method, apparatus and storage medium provided by the present application have the following technical effects:

[0021] The application first determines configuration information of a target format of a log; the configuration information includes at least one structure node, and each structure node includes at least one entry node; then the entry nodes in each structure node are traversed to obtain an entry node set corresponding to each structure node; the entry node set corresponding to each structure node is converted into a member variable set of a target programming language corresponding to each structure node; thus the defined log format is converted into the member variable set of the target programming language; the member variable set of the target programming language corresponding to each structure node is called to generate a log object, and the log object is converted into a log data stream; thus the server does not need to directly read and write the target format of the initial log, and the log data stream can be directly generated according to the converted member variable set of the target programming language, manual intervention in the log definition format file conversion process is reduced, and the log format conversion accuracy is improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art and the advantages thereof, a brief introduction will be given to the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0023] Figure 1 is a schematic diagram of a log data stream generation system provided by an embodiment of the present application;

[0024] Figure 2 is a flowchart of a log data stream generation method provided by an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of target format configuration information of an XML log provided by an embodiment of the present application;

[0026] Figure 4 is Golang code converted from the target format configuration information of the XML log provided by an embodiment of the present application;

[0027] Figure 5 is a log data stream obtained by Golang code in Figure 4 is a log data stream obtained by Golang code in

[0028] Figure 6 is a work flowchart of a log reporting module provided by an embodiment of the present application;

[0029] Figure 7 is a framework diagram of a log data stream generation tool provided by an embodiment of the present application;

[0030] Figure 8is a structural schematic diagram of a log data stream generation device provided by an embodiment of the present application;

[0031] Figure 9 is a structural schematic diagram of a server provided by an embodiment of the present application;

[0032] Figure 10 is a structural schematic diagram of a blockchain system provided by an embodiment of the present application;

[0033] Figure 11 is a block structure schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0036] Cloud storage is a new concept extended and developed on the basis of the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that cooperatively provides data storage and business access functions by collecting a large number of various types of storage devices (storage devices are also referred to as storage nodes) in a network through cluster application, grid technology, and distributed storage file system functions, and the like.

[0037] At present, the storage method of the storage system is: creating a logical volume, and allocating a physical storage space for each logical volume when creating the logical volume, which may be a certain storage device or a disk group composed of several storage devices. The client stores data on a certain logical volume, that is, stores data on a file system, the file system divides the data into many parts, each part is an object, the object contains not only data but also additional information such as data ID (ID entity), and the file system writes each object into the physical storage space of the logical volume, and records the storage location information of each object, so that when the client requests to access data, the file system can enable the client to access the data according to the storage location information of each object.

[0038] The process of allocating the physical storage space for the logical volume by the storage system is: according to the capacity estimation of the object stored in the logical volume (the estimation often has a large margin relative to the actual capacity of the object to be stored) and the group of the redundant array of independent disk (RAID), the physical storage space is divided into strips in advance, and a logical volume can be understood as a strip, so that the logical volume is allocated with the physical storage space.

[0039] The terms involved in the present specification are explained as follows:

[0040] Python: a widely used scripting programming language in the field of computer, used for developing a log escaping tool;

[0041] UDP Internet: a connectionless transport protocol supported by the protocol set, which is called User Datagram Protocol (UDP).

[0042] XML: Extensible Markup Language, a subset of Standard Generalized Markup Language, abbreviated as XML; a markup language used to mark electronic files to have structural properties.

[0043] MySQL: a relational database management system.

[0044] Golang: a parallel programming language used by the server to actually implement.

[0045] Goroutine: a lightweight thread in the Golang language.

[0046] Please refer to Figure 1 , Figure 1 is a schematic diagram of a log data stream generation system provided by the embodiments of the present application, as shown in Figure 1As shown, the log data stream generation system can at least include a server 01 and a client 02.

[0047] Specifically, in the embodiments of the present specification, the server 01 can include a standalone server, or a distributed server, or a server cluster composed of multiple servers. The server 01 can include a network communication unit, a processor, a memory, and the like. Specifically, the server 01 can be configured to escape the target format of the log and generate a log data stream.

[0048] Specifically, in the embodiments of the present specification, the client 02 can include an entity device such as a smart phone, a desktop computer, a tablet computer, a notebook computer, a digital assistant, a smart wearable device, and the like. The client 02 can also include software running in the entity device, such as a web page provided by a service provider to a user, or an application provided by the service provider to the user. Specifically, the client 02 can be configured to query the generated log data stream online.

[0049] The following describes a log data stream generation method according to an embodiment of the present specification, Figure 2 is a flowchart of a log data stream generation method according to an embodiment of the present specification. The present specification provides the method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one of the many execution orders, and does not represent the only execution order. In actual system or server product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, the method can be applied to a server, such as Figure 2 As shown, the method can include:

[0050] S201: determining configuration information of a target format of a log; the configuration information includes at least one structure node, and each structure node includes at least one entry node.

[0051] In the embodiments of the present specification, as Figure 3As shown, the target format of the log can be an editable format based on XML, the structural node can be a struct node, and the entry node can be an entry node; each struct node 03 includes a struct tag 04, which is used to identify the struct node; each struct tag represents a definition of a log format. Each entry tag represents a field in the log. The name tag represents the name, the desc tag represents the corresponding description, the type tag represents the corresponding data type, and the size tag represents the data length. In a specific embodiment, for a game server, the field tag includes the time of the game time, the XML file is converted into a Golang data file on the game server side, and is converted into a MySQL field description file in a log storage service process. The log format can be edited by adding, deleting, and modifying the XML file, and the log and specific log fields can be added or deleted.

[0052] In the embodiments of the present specification, the target format of the self-defined nested structure log can also be used.

[0053] S203: Traverse the entry nodes in each structural node to obtain a set of entry nodes corresponding to each structural node.

[0054] Specifically, in the embodiments of the present specification, the target format of the log can include one or more struct nodes, and the set of entry nodes corresponding to a structural node can include one or more entry nodes. In a specific embodiment, as shown in the following table, a struct node includes four entry nodes. Figure 3

[0055] S205: Convert the set of entry nodes corresponding to each structural node into a set of member variables of a target programming language corresponding to each structural node.

[0056] Specifically, in the embodiments of the present specification, the set of member variables of the target programming language can be in the form of Golang code, so that the server can be called. One set of entry nodes includes one or more entry nodes, one set of member variables includes one or more entry nodes, the number of elements in the set of entry nodes is the same as the number of elements in the corresponding set of member variables; the custom field of the target programming language includes first identification information, each member variable after conversion includes second identification information, the first identification information is different from the second identification information, and is used to distinguish the custom field of the target programming language and the member variable. For example, when the target programming language is Golang, the custom field of Golang starts with a lowercase m, and the member variable of Golang can start with an uppercase M, so as to distinguish the two. The custom field of Golang can be meta information, such as the original struct name.​

[0057] In one specific embodiment, the converted Golang code is as shown in Figure 4 Figure 4 Figure 3 the target format configuration information of the XML log.

[0058] In the embodiments of the present specification, the method further comprises:

[0059] determining a first data type of each entry node in the target format of the log;

[0060] determining a second data type of each member variable in the member variable set of the target programming language;

[0061] building a mapping relationship between the first data type and the second data type;

[0062] storing the mapping relationship between the first data type and the second data type based on a preset database;

[0063] In the embodiments of the present specification, the first data type and the second data type can be the same data type or different data types.

[0064] In the embodiments of the present specification, the converting the entry node set corresponding to each structure node into the member variable set of the target programming language corresponding to each structure node comprises:

[0065] converting a target entry node in the entry node set corresponding to each structure node into a target member variable in the member variable set of the target programming language corresponding to each structure node.

[0066] In the embodiments of the present specification, the converting the target entry node in the entry node set corresponding to each structure node into the target member variable in the member variable set of the target programming language corresponding to each structure node comprises:

[0067] determining a target entry node in the entry node set corresponding to each structure node;

[0068] determining a first target data type corresponding to the target entry node;

[0069] finding a second target data type matching the first target data type from the preset database;

[0070] converting the target entry node into a target member variable of the target programming language, the target member variable being of the second target data type.

[0071] In the embodiments of the present specification, the method further comprises: ​​

[0072] setting a meta tag for each member variable in a member variable set of the target programming language;

[0073] In the embodiments of the present specification, the meta tag is a meta Tag. The setting of the meta tag can associate the name defined in the original XML log with the member variable of Golang, and is mainly used for the use of the reflection mechanism of Golang.

[0074] After the step of converting the target entry node into the target member variable of the target programming language, the method further comprises:

[0075] determining a target meta tag of the target member variable;

[0076] based on the target meta tag of the target member variable, constructing a mapping relationship between the target entry node and the target member variable.

[0077] Specifically, in the embodiments of the present specification, Python can be used as a conversion tool for member variables; when the tool is used for conversion, the path of the XML file and the path of the converted Golang file need to be input; when the conversion is performed, all struct nodes in the XML file are traversed, and the definitions of the struct nodes are converted into classes (structs) in Golang in turn. In each struct node in the XML file, each entry node is traversed in turn, and the entry node is converted into a member variable of a corresponding data type. The converted member variable starts with a capital M and is distinguished from a Golang custom field starting with a lowercase m (these custom fields can be some meta information, such as the original struct name). All Golang member variables also generate corresponding tags (meta Tag), which associate the name defined in the original XML with the member variable of Golang, and are mainly used for the use of the reflection mechanism of Golang.

[0078] In one specific embodiment, the mapping relationship between the data types in the target format definition of the XML log and the data types of Golang is shown in Table 1.

[0079] Table 1: Mapping relationship between data types in the target format definition of the XML log and Golang data types

[0080]

[0081]

[0082] S207: Call the member variable set of the target programming language corresponding to each struct node to generate a log object.

[0083] In the embodiments of the present specification, the server generates a corresponding log object by calling a set of member variables of a target programming language corresponding to each structure node as a template.

[0084] S209: converting the log object into a log data stream.

[0085] Specifically, in the embodiments of the present specification, the fields in the log data stream and the fields in the configuration information of the target format of the log are in a one-to-one correspondence.

[0086] In the embodiments of the present specification, after the step of converting the log object into a log data stream, the method further comprises:

[0087] verifying the log data stream based on a preset data verification rule;

[0088] When there is error information in the log data stream, deleting the log data stream.

[0089] In the embodiments of the present specification, for the Golang native code automatically converted from the log, more data interfaces are provided, such as data verification rules, etc., which can verify some error key log generation, such as date, format, etc. For example, the time of the log is greatly different from the current time, or the user ID is 0, which can be determined as an error log.

[0090] In the embodiments of the present specification, the ToString method is used to convert a Golang log object into a log text of a data stream; compared with serialization and deserialization using Protobuf and other external data protocols, this conversion method is more efficient.

[0091] In one embodiment, a tool using the Golang reflection mechanism to convert a Golang object into a data stream is implemented by simulating Protobuf. The tool is run 100000 times on an experimental machine (8 cores, 32G memory), and the results are shown in Table 2. Compared with the conversion method of the present application and other solutions (such as Protobuf) using the reflection mechanism, it can be seen that the conversion method of the present application consumes less time and has a faster conversion speed.

[0092] Table 2:

[0093] Time taken to run 100000 data conversions Data stream conversion method of the present application 186.612077ms Data stream conversion method using reflection mechanism 771.110668ms

[0094] According to Figure 4 The log data stream obtained from the Golang code in the middle is shown in Figure 5 The "beFollowedNum" in the header is the noun of the log, and the numerical part is the specific data content, separated by "|", Figure 5The middle field corresponds to Figure 3 The configuration information in the middle target format, Figure 5 The "beFollowedNum" in the middle corresponds to Figure 3 The user attention table in the middle, Figure 5 The "2020-01-21" in the middle corresponds to Figure 3 The time of the game event in the middle; Figure 5 The "35077854" in the middle corresponds to Figure 3 The helper ID in the middle; Figure 5 The "52" in the middle corresponds to Figure 3 The number of attentions in the middle; Figure 5 The "1" in the middle corresponds to Figure 3 The test environment identifier in the middle.

[0095] In the embodiments of the present specification, after the step of converting the log object into a log data stream, the method further comprises:

[0096] Storing the log data stream in a buffer pool;

[0097] Determining the memory value of the newly added log data stream in the buffer pool within a preset time period;

[0098] When the memory value of the newly added log data stream is greater than a preset threshold, deleting the newly added log data stream.

[0099] In the embodiments of the present specification, when the newly added logs generated within a preset time period are too many, the log reporting thread will cause congestion, and deleting the newly added logs can ensure the normal operation of log uploading.

[0100] In the embodiments of the present specification, after the step of determining the memory value of the newly added log data stream in the buffer pool within a preset time period, the method further comprises:

[0101] Setting a preset threshold for the memory according to the storage space of the buffer pool;

[0102] Determining whether the memory value of the newly added log data stream in the buffer pool is greater than the preset threshold;

[0103] When the memory value of the newly added log data stream is less than or equal to the preset threshold, storing the newly added log data stream in the buffer pool.

[0104] In the embodiments of the present specification, the buffer pool can be used to prevent log congestion, thereby ensuring the normal operation of log storage.

[0105] In the embodiments of the present specification, the method further comprises:

[0106] Storing the log data stream in the buffer pool in a preset log database.

[0107] In the embodiments of the present specification, the method further comprises:

[0108] The log data stream in the buffer pool is stored based on a blockchain system. The blockchain system comprises a plurality of nodes, and a peer-to-peer network is formed among the plurality of nodes.

[0109] In some embodiments, the blockchain system can be Figure 10 The structure shown, a plurality of nodes form a composed peer-to-peer (P2P, Peer To Peer) network among them, and the P2P protocol is an application layer protocol running on the transmission control protocol (TCP, Transmission Control Protocol) protocol. In the blockchain system, any machine such as a server, a terminal can join to become a node, and the node includes a hardware layer, an intermediate layer, an operating system layer and an application layer.

[0110] Figure 10 The functions of the nodes in the blockchain system shown, the functions involved include:

[0111] 1) Routing, a basic function of the node, used to support communication between nodes.

[0112] In addition to the routing function, the node can also have the following functions:

[0113] 2) Application, used to be deployed in the blockchain, to implement specific business according to actual business needs, to record data related to the implementation function to form record data, to carry a digital signature in the record data to represent the source of the task data, and to send the record data to other nodes in the blockchain system. When the other nodes successfully verify the source and integrity of the record data, the record data is added to the temporary block.

[0114] 3) Blockchain, including a series of blocks (Block) connected to each other in the order of time of generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the node in the blockchain system.

[0115] In some embodiments, the block structure (Block Structure) can be Figure 11 The structure shown, each block includes the hash value of the transaction record stored in the block (the hash value of the block), and the hash value of the previous block, and the blocks are connected by the hash value to form a blockchain. In addition, the block can also include information such as the timestamp when the block is generated. The blockchain (Block chain) is essentially a decentralized database, which is a series of data blocks associated using cryptographic methods. Each data block contains relevant information for verifying the validity (anti-fake) of its information and generating the next block.

[0116] In the embodiments of the present specification, the setting of the buffer pool can effectively avoid log congestion in the preset log database.

[0117] Specifically, in the embodiments of the present specification, a log reporting module is embedded in the server code, which is a function module running in an independent thread, receives the log generated by the business side through chan (a pipeline data structure for communication between different Goroutines in Golang), and transmits it to the data center through a UDP connection. The workflow is as shown in Figure 6 , wherein the game business thread and the log reporting thread belong to the same process, both in the game server, and the data center can be in the game server or in other servers outside the game server; the game business thread calls the Golang code to generate the key log and delivers the log to the log reporting thread; the log reporting thread obtains the IP address and port of the data center and parses the UDP address to establish a connection with the UDP port in the data center process. After the connection is successful, the key log is delivered, the data stream log is obtained, and the data stream log is sent to the data center process; the data center process receives the data stream log and stores it in the corresponding log landing database (i.e. log landing database, also known as log storage database).

[0118] In the embodiments of the present specification, the buffer pool of the chan can be used for congestion control. If too many key logs are generated within a preset time period or at the same time, the chan will time out, and the generated key logs will be discarded and recorded in the system error log. When writing to the UDP connection, if an error is encountered or the timeout, the log reporting module will reattempt the complete UDP connection process, including: (1) reading the IP address and port of the data center (ip:port) from the process configuration information; (2) parsing the UDP address according to ip:port; (3) attempting to establish a connection until the connection is successful. Figure 6

[0119] In the embodiments of the present specification, the data center is a plurality of independent running processes that listen to the corresponding UDP port and receive the key log reporting from the game business side. According to the log type defined by XML, the log is stored in the corresponding database. The key log can include multiple types, such as stream log, login log, etc.

[0120] In the embodiments of the present specification, the storing of the log data stream in the buffer pool in the preset log database comprises:

[0121] determining at least one log corresponding to the log data stream in the buffer pool;

[0122] ​determine type identification information corresponding to each log;

[0123] determine a log database corresponding to each log according to the type identification information corresponding to each log;

[0124] store each log in the log database corresponding to each log.

[0125] Specifically, in the embodiments of the present specification, logs can be classified according to log types, for example, into water flow logs, login logs, etc., and a mapping relationship database of log type identification and database identification information is constructed. The determination of the log database corresponding to each log according to the type identification information corresponding to each log can include:

[0126] According to the type identification information corresponding to each log, the identification information of the database corresponding to the type identification information corresponding to each log is found from the mapping relationship database, so as to determine the log database corresponding to each log.

[0127] In the embodiments of the present specification, logs are classified and stored, thereby facilitating subsequent user search.

[0128] In the embodiments of the present specification, the method further includes:

[0129] obtain an Internet protocol address and port information of a target server;

[0130] determine a user datagram protocol address based on the Internet protocol address and port information;

[0131] send log data streams in the buffer pool to the target server based on the user datagram protocol address.

[0132] In the embodiments of the present specification, the log storage database can be set in the target server, and the target server includes a data center process; the data center process is provided with a UDP port. During log uploading, complex fault tolerance and completeness functions can be provided, such as token buffering and multi-path backup reporting.

[0133] In a specific embodiment, for example, three logs to be transmitted are set in a token, but only two logs are transmitted during log uploading, indicating that the logs are missing, so that the token buffering can timely find problems in the log uploading process, improve the completeness of the log uploading, and reduce the error rate of the log uploading.

[0134] Specifically, in the embodiments of the present specification, as shown in Figure 7 Figure 7 ​The framework diagram of the tool for generating log data stream is generated, wherein the log can be a key log, an interface provided by the tool to the outside is seen from the "XML log format definition", an external person modifies the XML log format definition, and then synchronizes the XML log to a game server and a data center (namely, a separate process set for storing the key log), so as to realize the updating operation of the log format definition.

[0135] The "XML log definition conversion tool" is used for converting the defined XML file into Golang code, which includes two parts: "data structure generation" and "data interface generation". The "data structure generation" converts the struct node in the XML into an actual Golang struct type (a kind of Golang native data format), and performs data format mapping, such as integer type, string type and the like. The "data interface generation" generates the method of the Golang struct, such as the ToString() method, which can convert a specific Golang log object into streaming data.

[0136] In the server, the key log is generated by specific business logic. In the tool framework, a separate key log reporting module is provided for congestion control and remote connection management. The module communicates with the data center through UDP, synchronizes the key log in real time, and stores it in the database (MySQL 1-N represents multiple database tables). In the storage process, the key log can be classified according to the type of the key log, the mapping relationship between the log type and the database is constructed in advance, and different key logs are stored in the corresponding database according to the category.

[0137] Specifically, in the embodiment of the present specification, in the XML log format, the struct node is used as the definition of a log, and each entry node represents a specific field. After modifying the XML, the XML log definition conversion tool needs to be called to convert the XML into Golang code and synchronize it to the game server project. At the same time, it needs to be synchronized to the data center to modify the related definition of MySQL. The modification of the XML log needs to follow the compatibility principle of the data in the database.

[0138] In the embodiments of the present application, the defined key log format is directly converted into the original code of Golang, including data structure and data flow conversion interface. No external data protocol is used, which reduces the transplantation cost and reduces the additional performance overhead; the UDP protocol is used to report the lightweight data flow format log, and an independent process is used to receive the key log and store the database. The definition of the database storage and the original file definition are unified, which reduces the escape cost. The generation, reporting and storage of the log are completely decoupled, which has portability and scalability. During the server runtime, the XML file is not directly read and written, so the efficient work of the server can be guaranteed.

[0139] As can be seen from the technical solutions provided by the embodiments of the present application, the configuration information of the target format of the log is first determined; the configuration information includes at least one structure node, and each structure node includes at least one entry node; then the entry nodes in each structure node are traversed to obtain the entry node set corresponding to each structure node; the entry node set corresponding to each structure node is converted into the member variable set of the target programming language corresponding to each structure node; thereby the defined log format is converted into the member variable set of the target programming language; the member variable set of the target programming language corresponding to each structure node is called to generate a log object, and the log object is converted into a log data stream; thereby the initial log target format does not need to be directly read and written by the server, the log data stream can be directly generated according to the converted member variable set of the target programming language, the manual intervention in the log definition format file conversion process is reduced, and the accuracy of the log format conversion is improved.

[0140] The embodiments of the present application also provide a log data stream generation device, as shown in Figure 8 The device includes:

[0141] The configuration information determination module 810 is configured to determine the configuration information of the target format of the log; the configuration information includes at least one structure node, and each structure node includes at least one entry node;

[0142] The entry node set determination module 820 is configured to traverse the entry nodes in each structure node to obtain the entry node set corresponding to each structure node;

[0143] The member variable set determination module 830 is configured to convert the entry node set corresponding to each structure node into the member variable set of the target programming language corresponding to each structure node;

[0144] The log object generation module 840 is configured to call the member variable set of the target programming language corresponding to each structure node to generate a log object;

[0145] The log data stream generation module 850 is configured to convert the log object into a log data stream.

[0146] In some embodiments, the apparatus can further include:

[0147] The first data type determination module is configured to determine a first data type of each entry node in a target format of the log.

[0148] The second data type determination module is configured to determine a second data type of each member variable in a member variable set of the target programming language.

[0149] The first mapping relationship construction module is configured to construct a mapping relationship between the first data type and the second data type.

[0150] The mapping relationship storage module is configured to store the mapping relationship between the first data type and the second data type based on a preset database.

[0151] In some embodiments, the member variable set determination module can include:

[0152] The conversion unit is configured to convert a target entry node in a target entry node set corresponding to each structure node into a target member variable in a member variable set of a target programming language corresponding to each structure node.

[0153] In some embodiments, the conversion unit includes:

[0154] The target entry node determination subunit is configured to determine a target entry node in the entry node set corresponding to each structure node.

[0155] The first target data type determination subunit is configured to determine a first target data type corresponding to the target entry node.

[0156] The second target data type determination subunit is configured to find a second target data type matching the first target data type from the preset database.

[0157] The target member variable conversion subunit is configured to convert the target entry node into a target member variable of the target programming language, and the target member variable is of the second target data type.

[0158] In some embodiments, the apparatus can further include:

[0159] The meta tag setting module is configured to set a meta tag of each member variable in the member variable set of the target programming language.

[0160] The target meta tag determination module is configured to determine a target meta tag of the target member variable.

[0161] The second mapping relationship construction module is configured to construct a mapping relationship between the target entry node and the target member variable based on a target meta tag of the target member variable.

[0162] In some embodiments, the apparatus can further include:

[0163] The first log data stream storage module is configured to store the log data stream in a buffer pool.

[0164] The memory value determination module is configured to determine a memory value of a newly added log data stream in the buffer pool within a preset time period.

[0165] The deletion module is configured to delete the newly added log data stream when the memory value of the newly added log data stream is greater than a preset threshold.

[0166] In some embodiments, the apparatus can further include:

[0167] The second log data stream storage module is configured to store the log data stream in the buffer pool in a preset log database.

[0168] In some embodiments, the second log data stream storage module includes:

[0169] The log determination unit is configured to determine at least one log corresponding to the log data stream in the buffer pool.

[0170] The type identification information determination unit is configured to determine type identification information corresponding to each log.

[0171] The log database determination unit is configured to determine a log database corresponding to each log according to the type identification information corresponding to each log.

[0172] The log storage unit is configured to store each log in the log database corresponding to each log.

[0173] In some embodiments, the apparatus can further include:

[0174] The information acquisition module is configured to acquire an Internet protocol address and port information of a target server.

[0175] The protocol address determination module is configured to determine a user datagram protocol address based on the Internet protocol address and port information.

[0176] The log data stream sending module is configured to send the log data stream in the buffer pool to the target server based on the user datagram protocol address.

[0177] The apparatus in the apparatus embodiment and the method embodiment are based on the same inventive concept.

[0178] The embodiment of the present application provides a log data stream generation device, which comprises a processor and a memory, and the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to realize the log data stream generation method provided in the above method embodiment.

[0179] The embodiment of the present application also provides a computer storage medium, which can be arranged in a terminal to save at least one instruction or at least one program related to the log data stream generation method in the method embodiment, and the at least one instruction or at least one program is loaded and executed by the processor to realize the log data stream generation method provided in the above method embodiment.

[0180] Optionally, in the embodiment of the present application, the storage medium can be located in at least one network server in a plurality of network servers of a computer network. Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various storage program codes.

[0181] The memory in the embodiment of the present application can be used to store software programs and modules, and the processor executes various function applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by functions and the like; and the data storage area can store data created according to the use of the device and the like. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, at least one magnetic disk storage device, a flash memory device or other volatile solid-state memory device. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0182] The log data stream generation method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal, a server or similar computing devices. Taking the case of running on a server as an example, Figure 9 is a hardware structure block diagram of a server for a log data stream generation method provided in the embodiment of the present application. As shown in the figure, Figure 9As shown, the server 900 can vary greatly in configuration and performance, and can include one or more Central Processing Units (CPU) 910 (processor(s) 910 can include, without limitation, a microprocessor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a processor of a mobile device, etc.), memory 930 for storing data, one or more storage media 920 (e.g., one or more mass storage devices) for storing applications 923 or data 922. The memory 930 and the storage media 920 can be of any type known or to be developed, including short term memory or long term storage. The programs stored in the storage media 920 can include one or more modules, each of which can include a series of instructions for operating a server. Further, the CPU 910 can be configured to communicate with the storage media 920 to execute the series of instructions of the storage media 920 on the server 900. The server 900 can also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input / output interfaces 940, and / or one or more operating systems 921, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0183] The input / output interface 940 can be configured to receive or transmit data via a network. Examples of the network can include a wireless network provided by a communication provider of the server 900. In one example, the input / output interface 940 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the input / output interface 940 can be a radio frequency (RF) module that is configured to communicate with the Internet via a wireless manner.

[0184] Those of ordinary skill in the art can understand that, Figure 9 The structure shown is merely illustrative and does not limit the structure of the electronic device described above. For example, the server 900 can include more or fewer components than those shown, or have a different configuration of components than those shown. Figure 9 For example, the server 900 can include more or fewer components than those shown, or have a different configuration of components than those shown. Figure 9 For example, the server 900 can include more or fewer components than those shown, or have a different configuration of components than those shown.

[0185] According to the log data stream generation method, device or storage medium provided in the application, the configuration information of the target format of the log is determined first; the configuration information includes at least one structure node, and each structure node includes at least one entry node; then the entry nodes in each structure node are traversed to obtain a set of entry nodes corresponding to each structure node; the set of entry nodes corresponding to each structure node is converted into a set of member variables of a target programming language corresponding to each structure node; thus the defined log format is converted into the set of member variables of the target programming language; the set of member variables of the target programming language corresponding to each structure node is called to generate a log object, and the log object is converted into a log data stream; thus the target format of the initial log does not need to be directly read and written by the server, the log data stream can be directly generated according to the set of member variables of the target programming language after conversion, manual intervention in the log definition format file conversion process is reduced, and the log format conversion accuracy is improved.

[0186] It should be noted that the above-mentioned embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments. The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or advantageous.

[0187] Each embodiment in the present specification is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the device, equipment and storage medium embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0188] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program instructions to relevant hardware, and the program can be stored in a computer storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0189] The above is only the preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating log data streams, characterized in that, The method includes: Configuration information that determines the target format of the log; the configuration information includes at least one structure node, and each structure node includes at least one entry node; Traverse the entry nodes in each structural node to obtain the entry node set corresponding to each structural node; The set of entry nodes corresponding to each structural node is converted into the set of member variables of the target programming language corresponding to each structural node; Call the set of member variables of the target programming language corresponding to each structure node to generate a log object; The log object is converted into a log data stream, and the log data stream is reported to the data center; The log data stream is stored in a buffer pool; Determine the memory value of newly added log data streams in the buffer pool within a preset time period; When the memory value of the newly added log data stream exceeds a preset threshold, the newly added log data stream is deleted.

2. The method according to claim 1, characterized in that, The method further includes: Determine the first data type of each entry node in the target format of the log; Determine the second data type of each member variable in the member variable set of the target programming language; Establish a mapping relationship between the first data type and the second data type; The mapping relationship between the first data type and the second data type is stored in the pre-defined database. The step of converting the set of entry nodes corresponding to each structural node into the set of member variables of the target programming language corresponding to each structural node includes: The target entry node in the entry node set corresponding to each structural node is converted into the target member variable in the member variable set of the target programming language corresponding to each structural node.

3. The method according to claim 2, characterized in that, The step of converting the target entry node in the entry node set corresponding to each structural node into the target member variable in the member variable set of the target programming language corresponding to each structural node includes: Determine the target entry node in the entry node set corresponding to each structural node; Determine the first target data type corresponding to the target entry node; Search the preset database for a second target data type that matches the first target data type; The target entry node is converted into a target member variable of the target programming language, and the target member variable is a second target data type.

4. The method according to claim 3, characterized in that, The method further includes: Set the meta label for each member variable in the member variable set of the target programming language; After the step of converting the target entry node into a target member variable of the target programming language, the method further includes: Determine the target meta label for the target member variable; Based on the target meta-tags of the target member variables, a mapping relationship between the target entry node and the target member variables is constructed.

5. The method according to claim 1, characterized in that, The method further includes: The log data stream in the buffer pool is stored in a preset log database.

6. The method according to claim 5, characterized in that, The step of storing the log data stream in the buffer pool in a preset log database includes: Identify at least one log corresponding to the log data stream in the buffer pool; Determine the type identification information corresponding to each log; Based on the type identifier information corresponding to each log, determine the log database corresponding to each log; Each log entry is stored in the log database corresponding to that log entry.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the Internet Protocol address and port information of the target server; The user datagram protocol address is determined based on the Internet Protocol address and port information; Based on the User Datagram Protocol address, the log data stream in the buffer pool is sent to the target server.

8. A log data stream generation device, characterized in that, The device includes: The configuration information determination module is used to determine the configuration information of the target format of the log; the configuration information includes at least one structure node, and each structure node includes at least one entry node; The entry node set determination module is used to traverse the entry nodes in each structural node to obtain the entry node set corresponding to each structural node. The member variable set determination module is used to convert the entry node set corresponding to each structural node into the member variable set of the target programming language corresponding to each structural node. The log object generation module is used to call the set of member variables of the target programming language corresponding to each structure node to generate a log object; The log data stream generation module is used to convert the log object into a log data stream and report the log data stream to the data center; The first log data stream storage module is used to store the log data stream in a buffer pool; The memory value determination module is used to determine the memory value of newly added log data streams in the buffer pool within a preset time period; The deletion module is used to delete the newly added log data stream when the memory value of the newly added log data stream is greater than a preset threshold.

9. The apparatus according to claim 8, characterized in that, The device further includes: The first data type determination module is used to determine the first data type of each entry node in the target format of the log. The second data type determination module is used to determine the second data type of each member variable in the member variable set of the target programming language. The first mapping relationship construction module is used to construct the mapping relationship between the first data type and the second data type; the mapping relationship storage module is used to store the mapping relationship between the first data type and the second data type based on a preset database. The member variable set determination module includes: The conversion unit is used to convert the target entry node in the entry node set corresponding to each structure node into the target member variable in the member variable set of the target programming language corresponding to each structure node.

10. The apparatus according to claim 9, characterized in that, The conversion unit includes: The target entry node determination subunit is used to determine the target entry node in the entry node set corresponding to each structural node; The first target data type determination subunit is used to determine the first target data type corresponding to the target entry node; The second target data type determination subunit is used to search for a second target data type that matches the first target data type from the preset database; The target member variable conversion subunit is used to convert the target entry node into a target member variable of the target programming language, wherein the target member variable is a second target data type.

11. The apparatus according to claim 10, characterized in that, The device further includes: The meta tag setting module is used to set the meta tag for each member variable in the member variable set of the target programming language; The target meta-label determination module is used to determine the target meta-label of the target member variable; The second mapping relationship construction module is used to construct a mapping relationship between the target entry node and the target member variable based on the target meta tag of the target member variable.

12. The apparatus according to claim 8, characterized in that, The device further includes: The second log data stream storage module is used to store the log data stream in the buffer pool in a preset log database.

13. The apparatus according to claim 12, characterized in that, The second log data stream storage module includes: A log determination unit is used to determine at least one log corresponding to the log data stream in the buffer pool; The type identification information determination unit is used to determine the type identification information corresponding to each log. The log database determination unit is used to determine the log database corresponding to each log based on the type identification information corresponding to each log. A log storage unit is used to store each log in the log database corresponding to each log.

14. The apparatus according to claim 8, characterized in that, The device further includes: The information acquisition module is used to acquire the Internet Protocol address and port information of the target server; The protocol address determination module is used to determine the user datagram protocol address based on the Internet Protocol address and port information; The log data stream sending module is used to send the log data stream in the buffer pool to the target server based on the User Datagram Protocol address.

15. A log data stream generation device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction or at least one program, which is loaded and executed by the processor to implement the log data stream generation method as described in any one of claims 1-7.

16. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the log data stream generation method as described in any one of claims 1-7.

Images