Log data serialization method and device, equipment and medium

Abstract

The invention relates to a log data serialization method and device, equipment and a medium, and the method comprises the steps: positioning header information of log data based on a virtual address in a virtual address space corresponding to a physical storage region of the log data, and analyzing the header information to determine an organization structure type of the log data; when an organization structure of an event log stored in the log data is a flow structure type, performing byte-level analysis on the log data based on the virtual address, and performing serialization processing on a result obtained by byte-level analysis according to a preset serialization format; and when the organization structure is of a nested structure type, performing recursive byte-level analysis and serialization processing on the log data of the nested structure by adopting a dynamic adjustment stack mechanism. According to the method, the serialization efficiency of the log data is effectively improved through application of the virtual address and a byte level analysis technology in combination with a dynamic adjustment stack mechanism, and meanwhile memory occupation is reduced.

Description

Technical Field

[0001] The present application relates to the field of e-commerce information processing technology, and in particular to a log data serialization method and its device, equipment, and medium. Background Art

[0002] E-commerce platforms need to process massive amounts of user behavior data, transaction records, and product information during their operations. This data is recorded in the form of logs and converted into byte streams through serialization for efficient transmission and storage in appropriate storage media. When data analysis and mining on e-commerce platforms is required, the corresponding byte streams in the storage media are restored to the original log data through deserialization, such as analyzing user behavior, optimizing recommendation algorithms, and detecting abnormal transactions. However, existing log serialization methods often have many problems when processing massive amounts of data.

[0003] As e-commerce platforms continue to expand and their operations become increasingly complex, the rate and volume of log data generated are exploding. Traditional log serialization methods suffer from poor performance when processing this large-scale and complex log data due to frequent memory allocation and copying operations. For example, when processing nested log data, reflection can significantly increase the time overhead of serialization and deserialization, making it difficult to meet the real-time requirements of high-concurrency scenarios.

[0004] In addition, when processing nested log data, traditional log serialization methods may cause stack overflow and other problems during parsing and serialization because the stack depth is fixed. This makes it impossible to efficiently process deeply nested log data, which in turn affects the overall performance and stability of the e-commerce platform and leads to a decline in user experience. Summary of the Invention

[0005] The primary purpose of the present application is to solve at least one of the above problems and provide a log data serialization method and its device, equipment, and medium.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A log data serialization method provided to meet one of the purposes of this application includes the following steps:

[0008] Locating header information of the log data based on a virtual address in a virtual address space corresponding to a physical storage area of the log data, and parsing the header information to determine an organizational structure type of the log data;

[0009] When the organizational structure of the event log stored in the log data is a process structure type, performing byte-level parsing on the log data based on the virtual address, and serializing the result obtained by the byte-level parsing according to a preset serialization format;

[0010] When the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is adopted to perform recursive byte-level parsing and serialization processing on the log data of the nested structure.

[0011] On the other hand, a log data serialization device provided to meet one of the purposes of the present application includes:

[0012] a structure type determination module, configured to locate header information of the log data based on a virtual address in a virtual address space corresponding to a physical storage area of the log data, and parse the header information to determine an organizational structure type of the log data;

[0013] a process structure processing module, configured to, when the organizational structure of the event log stored in the log data is a process structure type, perform byte-level parsing on the log data based on the virtual address, and serialize the result obtained by the byte-level parsing according to a preset serialization format;

[0014] The nested structure processing module is used to adopt a dynamic stack adjustment mechanism to perform recursive byte-level parsing and serialization processing on the log data of the nested structure when the organizational structure is a nested structure type.

[0015] On the other hand, a computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the log data serialization method described in the present application.

[0016] On the other hand, a computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the log data serialization in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0017] The technical solution of this application has many advantages, including but not limited to the following:

[0018] This application realizes efficient serialization processing of massive log data by introducing byte-level parsing technology, virtual address space and dynamic stack adjustment mechanism, significantly improving processing efficiency and flexibility of memory management. Specifically, the physical storage area of the log data is mapped to the virtual address space, and the log data can be directly parsed at the byte level through the virtual address, which not only improves the security and flexibility of data access, but also significantly reduces memory copy operations, thereby greatly improving processing efficiency; when processing nested log data, this application can flexibly expand or shrink the stack capacity according to actual needs by dynamically adjusting the stack space, which not only avoids the risk of stack overflow, but also reduces memory waste through intelligent memory management.

[0019] In terms of processing efficiency, this application directly accesses log data through virtual addresses, avoiding multiple memory copies and significantly improving the access speed of log data. Secondly, in terms of memory management, the dynamic stack adjustment mechanism not only solves the stack overflow problem, but also reduces memory waste through intelligent memory allocation. This application provides significant performance improvements and resource optimization for scenarios such as e-commerce platforms that need to process massive amounts of log data. During the operation of e-commerce platforms, when log data such as user behavior data, transaction records, or system operation logs explode, these log data can be efficiently converted into byte streams through the log data serialization method of this application for easy storage and transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0021] Figure 1 This is a flowchart of a typical embodiment of the log data serialization method of the present application;

[0022] Figure 2 This is a functional block diagram of the log data serialization device of this application;

[0023] Figure 3 This is a schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0024] A log data serialization method of the present application can be programmed as a computer program product and deployed in a server for execution. For example, in the exemplary application scenario of the present application, it can be deployed in the server of an e-commerce platform, wherein the e-commerce platform can be an e-commerce platform that opens an independent station service. An independent station refers to a new type of official website (website) established on a SaaS technology platform with an independent domain name, private content, data, and rights, independent management sovereignty and management entity responsibility, supported by social cloud computing capabilities, and can independently and freely connect to third-party software tools, publicity and promotion media and channels. Usually, product sellers build and operate online stores on independent stations, allowing online stores to display their products on independent stations, and provide e-commerce services such as searching for products, adding shopping carts, settlement, logistics delivery, placing orders, and after-sales service to product buyers (i.e., users) on online stores. Product buyers (users) perform various interactive operations with the e-commerce platform on terminal devices. With the continuous expansion of the scale of e-commerce platforms and the improvement of business complexity, user behavior data, transaction records, system operation logs and other information have exploded. These data are recorded in the form of logs for subsequent data analysis and mining. Specifically, users of the e-commerce platform will generate a large amount of log data in various links such as browsing products, placing orders, paying and settling, and checking logistics. By analyzing these log data, the e-commerce platform can promptly discover potential problems, such as transaction anomalies, abnormal user behavior, system performance bottlenecks, etc., so as to quickly take measures to optimize and adjust. In this process, the serialization of this application can convert structured log data into a continuous byte stream form, which is convenient for storage in the file system, database or transmission over the network, thereby saving storage space and improving data transmission efficiency; at the same time, when it is necessary to analyze or reproduce log information, deserialization can restore the byte stream to the original log data structure to ensure the integrity and availability of the data. Through the serialization and deserialization process, it supports functions such as data analysis, mining and visualization of the e-commerce platform, helping the e-commerce platform to promptly discover and solve problems existing in the e-commerce platform, thereby optimizing operational efficiency and improving user experience.

[0025] In a typical embodiment of the present application, the organizational structure type of log data is first determined. Organizational structure types include stream structure types and nested structure types. The stream structure type refers to the log data being organized in a linear, continuous manner. Log records are arranged in chronological order, and each record is independent of each other without complex nested relationships. The stream structure type is suitable for recording simple behavioral events, such as user browsing behavior and click operations on e-commerce platforms. Each log record contains basic information such as timestamp, user ID, operation type, and operation object. These records are stored in the chronological order of the events, facilitating chronological analysis and processing. For example, when a user browses products on an e-commerce platform, each time a product page is clicked, the e-commerce platform generates a log record that records information such as the user ID, click time, and product ID. These records are arranged in a linear manner, forming a continuous log stream, which facilitates real-time monitoring and simple statistical analysis, such as counting user behavior frequency and analyzing user browsing paths. The nested structure type refers to the presence of multiple nested relationships in the log data, similar to a tree structure, containing multiple levels and branches. For example, a complex transaction process may consist of multiple sub-processes, each of which contains multiple steps, with a parent-child hierarchical relationship between these steps and sub-processes. Nested structures are suitable for documenting complex business processes, such as the order processing process on an e-commerce platform. Starting with a user placing an order, the entire process includes multiple steps, such as payment, order review, shipping, and logistics tracking. Each step is further subdivided into multiple sub-steps. For example, the payment process includes selecting a payment method, initiating a payment request, and whether the payment succeeds or fails. These sub-steps and steps are organized together through nested structures, forming a complex hierarchical relationship. This nested structure clearly illustrates the logical relationships and execution order between each step, facilitating in-depth analysis and debugging of complex business processes, such as tracking abnormal steps in order processing and analyzing the time consumption of each step.

[0026] After determining the organizational structure type of the log data, the log data is serialized according to different methods. In some embodiments, for log data of a process structure type, based on the virtual address in the virtual address space corresponding to the physical storage area of the log data, the log data is located and the log data is parsed at the byte level, the specific content of each log record is extracted, and the parsed result is serialized into a byte stream according to a preset serialization format (such as JSON, Protocol Buffers, etc.). For log data of a nested structure type, the root node of the nested structure is located based on the virtual address, and a stack for storing context information of the parsed nested level is initialized. Nested structures are parsed byte-by-byte, layer by layer, according to a depth-first traversal. When a hierarchical start feature is detected, a dynamic stack adjustment mechanism is implemented, and the parsed nested hierarchical context information is pushed onto the stack. When a preset backtracking condition is met, such as recognizing a JSON closing bracket "}" or a specific end identifier, the corresponding nested hierarchical context information (including metadata such as node type, substructure length, and parent node virtual address pointer) is popped from the stack and serialized according to a preset serialization format (such as Protocol Buffers' Tag-Length-Value encoding). This parsing and serialization process is iterated until the stack is empty, generating a complete nested structure serialization result.

[0027] In some embodiments, the serialized byte stream will be stored in a persistent storage medium, such as a disk file system, a distributed storage system (such as HDFS, Ceph) or a database (such as MySQL, MongoDB), etc., for subsequent reading and processing. When log data needs to be used for analysis, the corresponding byte stream is read from the storage medium and deserialized. For the specific deserialization steps, please refer to the subsequent specific implementation methods, which will not be repeated here. The deserialized log data can be used in a variety of analysis scenarios. By analyzing the user's browsing path, click behavior, purchase conversion rate, etc., the user experience and recommendation algorithm can be optimized; for example, by detecting abnormal behavior during the transaction process, such as payment failure, order anomalies, etc., transaction security can be guaranteed; by analyzing system operation logs, performance bottlenecks can be detected and system performance can be optimized. In addition, data mining technology can be used to discover hidden patterns and association rules in log data to provide support for business decision-making.

[0028] The log data serialization method of the present application can be programmed into a computer program product and deployed in a server for execution.

[0029] See also Figure 1 The log data serialization method of the present application, in its typical embodiment, includes the following steps:

[0030] Step S5100: Locate header information of the log data based on a virtual address in a virtual address space corresponding to a physical storage area of the log data, and parse the header information to determine an organizational structure type of the log data;

[0031] E-commerce platforms generate massive amounts of log data during their operations. The generated log data is converted into byte streams through serialization and stored in physical storage media (such as hard disks, solid-state drives, etc.). During this serialization process, it is necessary to determine the organizational structure type of the log data and adopt different serialization methods based on the organizational structure type.

[0032] In one embodiment, the e-commerce platform maps the physical storage area to the virtual address space through the memory management unit (MMU), so that the e-commerce platform can directly access the log data stored in the physical storage area through the virtual address. The use of the virtual address space not only improves the security of data access, but also makes memory management more flexible and efficient.

[0033] In the serialization process of log data, since different organizational structure types (such as process structure and nested structure) require different serialization methods, it is necessary to first determine the organizational structure type of the log data. In one embodiment, the organizational structure type of the log data is stored in the header information of the log data. The header information is the starting part of the log data file or data block, and contains meta-information about the log data, such as data format, version number, organizational structure type, etc. This step uses a virtual address to locate the header information of the log data. Specifically, through the page table mapping relationship of the memory management unit, the address of the physical storage area of the log data is converted into a virtual address to ensure that the virtual address can be correctly mapped to the corresponding physical storage area. After obtaining the virtual address of the log data header information, the header information of the corresponding log data is parsed based on the virtual address. The header information is stored in a specific format. For example, the header information contains a fixed-length header area, in which a field is set to identify the organizational structure type of the log data. The field is a simple flag bit, or a more complex encoding structure used to describe the organizational structure type of the log data.

[0034] This step directly accesses the header information of the log data stored in the physical address area based on the virtual address in the virtual address space, and parses the organizational structure type field to determine the specific method of subsequent serialization operations. The implementation of this step avoids multiple memory copies and significantly improves processing efficiency.

[0035] Step S5200: When the organizational structure of the event log stored in the log data is a process structure type, byte-level parsing is performed on the log data based on the virtual address, and a result obtained by the byte-level parsing is serialized according to a preset serialization format;

[0036] In the process of log data processing, process structure type log data is used to record simple behavioral events such as user operations and system status changes due to its linear and continuous characteristics. Process structure type log data is arranged in chronological order, and each log record is independent and has a relatively fixed format.

[0037] In this step, the log data is directly parsed at the byte level and serialized according to a preset serialization format. In one embodiment, the starting position of the log data is located by a virtual address. After locating the starting position of the log data, the log data is read byte by byte. During the reading process, the fields of each log record are parsed one by one according to a predefined format. Each log record contains multiple fields, such as timestamp, user ID, operation type, operation object, etc. Serialization refers to the conversion of structured data into a continuous byte stream for easy storage in a file system, database, or transmission over a network. Common serialization formats include JSON, Protocol Buffers, XML, etc. In this step, the parsed structured data is serialized according to a preset serialization format. The serialization process first selects a suitable serialization format. For example, if efficient storage and transmission are required, select Protocol Buffers; if easy reading and debugging are required, select JSON. After selecting the serialization format, the parsed structured data object is converted into a serialized object.

[0038] In one embodiment, the generated byte stream can be stored in a file system, a database, or transmitted to another system via a network. The specific storage or transmission method depends on the application scenario and user needs. For example, the serialized byte stream can be stored in a disk file system for subsequent reading and processing, or it can be transmitted over a network to another server or service for real-time analysis and monitoring.

[0039] Step S5300: When the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is adopted to perform recursive byte-level parsing and serialization processing on the log data of the nested structure.

[0040] The log data of the nested structure has a multi-layer nested relationship, similar to a tree structure, where each node contains one or more child nodes. The nested structure is used to record complex business processes (such as the order processing process of an e-commerce platform). In one embodiment, when it is determined that the organizational structure of the log data is a nested structure type, the root node of the nested structure is first located based on the virtual address, and the entire nested structure is parsed layer by layer from the root node, and the specific content of each level is parsed according to the predefined format. During the parsing process, the level starting feature is continuously detected. When the level starting feature is detected, it is determined to be a new nested level, and the level is parsed to identify the key information such as the level identifier, event type, event description, and number of sub-events of each node. Furthermore, after dynamically adjusting the stack space, the nested level context information obtained by parsing is pushed into the stack, and the required stack frame size is estimated based on the number of sub-events of the current nested level. If the remaining capacity of the current stack space is not enough to add a new stack frame, the stack space is automatically expanded to ensure that there is enough space to store the new context information, thereby avoiding parsing failure due to insufficient stack space. During the parsing of sub-events, the above process is repeated. Whenever a new nesting level start feature is identified, the stack space is dynamically adjusted and new context information is pushed onto the stack. When a backtracking condition is detected, such as when a nesting level end marker feature is identified, the context information for the current nesting level is popped from the stack. The last-in, first-out nature of the stack ensures that the context information popped is the most recently pushed onto the stack, that is, the context information for the nesting level currently being processed. This context information includes key information such as the current level's level identifier, event type, event description, number of sub-events, and a pointer to the parent level. This recursive parsing approach naturally handles the complexity of nested structures, ensuring that data at each level is correctly parsed. Serialization is the process of converting the parsed structured data into a continuous byte stream. During serialization, the log data is encoded into a byte stream based on the log data for the current nesting level and a preset serialization format. Simultaneously, the data is correctly nested and organized based on the hierarchical relationships in the context information to ensure that the serialized data accurately reflects the original nested structure. By executing the above parsing and serialization process, the log data of the nested structure is processed step by step until the stack is empty, and finally a complete nested structure serialization result is generated.

[0041] It can be seen from the typical embodiments of the present application that the technical solution of the present application has many advantages, including but not limited to the following aspects:

[0042] This application realizes efficient serialization processing of massive log data by introducing byte-level parsing technology, virtual address space and dynamic stack adjustment mechanism, significantly improving processing efficiency and flexibility of memory management. Specifically, through the page table mapping mechanism of the memory management unit, the physical storage area of the log data is mapped to the virtual address space, and the log data can be directly parsed at the byte level through the virtual address, which not only improves the security and flexibility of data access, but also significantly reduces memory copy operations, thereby greatly improving processing efficiency; when processing nested log data, this application can flexibly expand or shrink the stack capacity according to actual needs by dynamically adjusting the stack space, which not only avoids the risk of stack overflow, but also reduces memory waste through intelligent memory management.

[0043] In terms of processing efficiency, this application directly accesses log data through virtual addresses, avoiding multiple memory copies and significantly improving the access speed of log data. Secondly, in terms of memory management, the dynamic stack adjustment mechanism not only solves the stack overflow problem, but also reduces memory waste through intelligent memory allocation. This application provides significant performance improvements and resource optimization for scenarios such as e-commerce platforms that need to process massive amounts of log data. During the operation of e-commerce platforms, when log data such as user behavior data, transaction records, or system operation logs explode, these log data can be efficiently converted into byte streams through the log data serialization method of this application for easy storage and transmission.

[0044] In a further embodiment, based on a virtual address in a virtual address space corresponding to a physical storage area of the log data, locating the header information of the log data, and parsing the header information to determine the organizational structure type of the log data, the following steps are included:

[0045] Step S6100: Encode the organizational structure type of the log data into metadata in a preset format and write it into the type identification field of the log data header information;

[0046] During the serialization of log data, the organizational structure type of the event log stored in the log data must first be clearly identified so that subsequent parsing and processing can quickly identify the structural characteristics of the event log. This step encodes the organizational structure type of the log data into metadata in a preset format and writes it into the type identification field of the log data header information. In one embodiment, a preset format is designed, which can be a simple flag or a more complex encoding structure, to describe the organizational structure type of the log data. For example, a single byte can be used to represent the structure type, where "0x01" indicates a process structure type and "0x02" indicates a nested structure type. This encoding method is simple and easy to parse, allowing for quick identification of the organizational structure type of the log data. When the log data is generated, the corresponding code is written into the type identification field of the log data header information based on the actual structure type of the log data. The log data header information is the beginning of the log data file or data block and contains metadata about the log data, such as the data format, version number, and organizational structure type. By encoding the structure type code in the header information, subsequent serialization and parsing processes can directly determine the organizational structure type of the log data by reading the header information.

[0047] In another embodiment, in order to prevent parsing failures caused by data corruption or format errors, a checksum or redundant information is added to the header information. During the parsing process, the correctness of the checksum or redundant information is first verified. After confirming that the data is complete and correct, the organizational structure type field is parsed. In addition, this embodiment uses a caching mechanism to improve parsing efficiency. Specifically, after the header information of a log data file is parsed for the first time, the parsing results are cached. In subsequent serialization operations, if the organizational structure type of the log data needs to be determined again, the result can be obtained directly from the cache without re-reading and parsing the header information. This caching mechanism can significantly reduce the number of accesses to the physical storage, thereby improving overall performance.

[0048] Through the above steps, the organizational structure type of the event log is accurately encoded as metadata in a preset format and written into the type identification field of the log data header information, providing clear type determination for subsequent serialization and parsing, and ensuring processing efficiency and accuracy through data verification and caching mechanisms.

[0049] Step S6200: Based on the page table mapping relationship of the memory management unit, the starting address and length of the physical storage area where the log data is located are mapped to the virtual address space, and a corresponding virtual address is generated to support parsing the header information of the corresponding log data according to the virtual address to determine the structure type of the log data.

[0050] The memory management unit (MMU) is a key component responsible for managing the mapping between the virtual address space and the physical address space. Through the page table mapping mechanism of the MMU, the log data in the physical storage area can be mapped to the virtual address space, thereby achieving efficient access and processing of the log data. Specifically, it is first necessary to obtain the starting address and length of the physical storage area where the log data is located. The physical storage area refers to a continuous address range on a hard disk, solid-state drive or other storage medium, in which the log data is stored. For example, assuming that the log data is stored in a specific partition of the hard disk, the starting address and length information of the partition can be obtained through the interface of the file system. In one embodiment, this information is expressed in bytes. For example, the starting address of the log data is the 1024th byte on the hard disk and the length is 1MB.

[0051] The address range of the physical storage area is mapped to the virtual address space using the page table mapping mechanism of the memory management unit (MMU). The MMU manages the mapping relationship between virtual addresses and physical addresses through the page table. The page table is a data structure that stores mapping entries between virtual addresses and physical addresses. Each mapping entry specifies a virtual address range and a corresponding physical address range.

[0052] In another embodiment, one or more page table entries are created to map the physical address range of the log data to a continuous area in the virtual address space. For example, assuming the physical address range of the log data is from 0x100000 to 0x1FFFFF (1MB), this range is mapped to 0x20000000 to 0x200FFFFF in the virtual address space. When accessing the log data through the virtual address, the MMU automatically converts the virtual address to the corresponding physical address, thereby enabling access to the log data.

[0053] In this embodiment, the organizational structure type of the log data is encoded as metadata in a preset format and written into the type identification field of the header information. At the same time, the page table mapping mechanism of the memory management unit (MMU) is used to map the physical storage area where the log data is located to the virtual address space. After the mapping is completed, the log data can be accessed through the generated virtual address. The use of virtual addresses allows e-commerce platforms to access log data stored on the hard disk just like accessing data in memory. This mapping mechanism not only improves the flexibility of data access, but also reduces the number of memory copies, thereby significantly improving overall performance.

[0054] In a further embodiment, when the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is adopted to perform recursive byte-level parsing and serialization processing on the log data of the nested structure, including the following steps:

[0055] Step S5310: Locate the root node of the nested structure based on the virtual address, and initialize a stack for storing context information for parsing the nested hierarchy;

[0056] When processing log data with a nested structure, it is first necessary to determine the starting point of the nested structure, that is, the root node. The root node is the entrance to the entire nested structure, and the entire nested structure can be parsed layer by layer through the root node. In the storage structure of the log data, the root node of the nested structure has a clear identifier, such as a specific tag or location information. In one embodiment, the header information of the log data is accessed through a virtual address to obtain the specific location of the root node. The virtual address is used to directly locate the starting position of the log data without multiple memory copies or complex address conversions. For example, assuming that the header information of the log data contains a field that specifies the offset of the root node of the nested structure, the actual position of the root node can be quickly located by reading the field and combining it with the virtual address.

[0057] After determining the location of the root node, a stack needs to be initialized to store the context information of each layer during the parsing process. A stack is a last-in-first-out (LIFO) data structure that is suitable for handling nested structures. The stack can conveniently manage hierarchical relationships in recursive calls. When initializing the stack, an empty stack instance is created and initial storage space is allocated for the stack. The initial size of the storage space can be initialized based on the expected nesting depth and the size of the context information of each layer. The context information includes key information such as the identifier of the current level, the starting position of the current level, and the length of the current level. This information is crucial for subsequent parsing and serialization processing. For example, the identifier of the current level can be a simple integer used to distinguish different levels; the starting position of the current level is a virtual address that points to the starting point of the current level data; the length of the current level indicates the size of the current level data, which is used to control the scope of parsing.

[0058] Through the above process, the root node of the nested structure can be accurately located based on the virtual address, and a stack for storing parsed nested hierarchical context information can be initialized, providing a solid foundation for subsequent recursive byte-level parsing and serialization processing, ensuring that the entire nested structure can be processed efficiently.

[0059] Step S5320: Parse the nested structure layer by layer at the byte level according to the depth-first traversal rule. When a level start feature is detected, dynamically adjust the stack space to push the nested level context information obtained by the parsing into the stack.

[0060] Depth-first traversal is a commonly used tree or graph traversal method. Starting from the root node, the search proceeds deeply along the current branch path until the end of the path is reached. The search then returns to the previous branch point and continues searching other branches. This traversal method is suitable for processing nested log data, as nested structures can be viewed as trees, where each node may contain multiple child nodes, and each child node may contain further child nodes. When parsing nested log data, the system first starts by reading the log data byte by byte, starting from the root node and parsing the specific content of each node according to a predefined format. During the parsing process, the system continuously detects the level start feature. The level start feature is a specific pattern or marker in the log data that indicates the beginning of a new nested level. For example, a nested level may be marked by a specific byte sequence or a special field value. When the level start feature is detected, it is identified as a new nested level and parsing begins for that level, identifying key information such as the level identifier, event type, event description, and number of sub-events for each node.

[0061] Furthermore, the stack space is dynamically adjusted to push the parsed nested level context information into the stack. Specifically, the required stack frame size is estimated based on the number of sub-events in the current nested level. If the remaining capacity of the current stack space is insufficient to accommodate the new stack frame, the stack space is automatically expanded to ensure that there is enough space to store the new context information, thereby avoiding parsing failures due to insufficient stack space. For specific expansion methods, please refer to the subsequent specific implementation methods, which will not be elaborated here. In the process of parsing sub-events, the above process is repeated. Whenever a new nested level starting feature is encountered, the stack space is dynamically adjusted and the new context information is pushed into the stack. This recursive parsing method can naturally handle the complexity of the nested structure and ensure that the data at each level can be correctly parsed.

[0062] This recursive byte-level parsing based on depth-first traversal and dynamic stack space adjustment allows for efficient processing of nested log data. This step correctly parses complex nested relationships and flexibly adapts to nested structures of varying depth and complexity, providing an accurate and efficient foundation for subsequent log data analysis and mining.

[0063] Step S5330: When the preset backtracking condition is met, the corresponding nested level context information is popped from the stack and serialized according to the preset serialization format;

[0064] The backtracking condition is set in advance by those skilled in the art. For example, when encountering the end marker of a nested level, the backtracking process is performed. That is, when encountering the end marker of a nested level during the depth-first traversal, the backtracking condition is triggered. The end marker can be a specific byte sequence, a field value, or a signal that meets certain logical conditions, indicating that the parsing of the current nested level has been completed. For example, when parsing a nested structure in JSON format, encountering a closed curly brace "}" can be regarded as a marker for the end of a level.

[0065] When it is detected that the backtracking condition is met, the context information of the current nested level is popped from the stack. This context information contains key information such as the level identifier of the current level, event type, event description, number of child events, and pointer to the parent level. Serialization is the process of converting parsed structured data into a continuous byte stream. Specifically, this conversion process is performed according to a preset serialization format. The choice of serialization format depends on application requirements and performance considerations. Common serialization formats include JSON, Protocol Buffers, XML, etc. Among them, the JSON format is easy to read and debug, and Protocol Buffers excels in storage and transmission efficiency.

[0066] During serialization, data is encoded into a byte stream based on the data content at the current nesting level and the preset serialization format. For example, if the serialization format is JSON, the data at the current level is converted into a JSON object or array, encoding the data fields, such as encoding string fields as JSON strings and integer fields as JSON numbers. Furthermore, data is correctly nested and organized based on the hierarchical relationships in the context information to ensure that the serialized data accurately reflects the original nested structure.

[0067] Step S5340: iteratively execute the above parsing and serialization process until the stack is empty, and generate a complete nested structure serialization result.

[0068] By iteratively executing the above parsing and serialization process, the nested log data is processed step by step until the stack is empty, and a complete nested structure serialization result is finally generated. An empty stack indicates that the data of all nested levels has been parsed and serialized. At this point, the generated serialization result is a complete nested structure that accurately reflects the nested relationship and content of the original log data. This iterative processing method can efficiently process complex nested log data. For example, the order processing process of an e-commerce platform contains multiple nested levels, such as orders, payments, and shipments. Each level includes corresponding event types and sub-events. By iteratively executing the parsing and serialization process, each level can be processed step by step, and finally a complete nested structure serialization result is generated, which can be used for subsequent data analysis, monitoring, or storage.

[0069] In addition, the iterative processing method of this embodiment can dynamically adjust the parsing and serialization logic according to different serialization formats and requirements. For example, if a new serialization format needs to be supported, it is only necessary to add the corresponding encoding logic in the serialization processing stage. At the same time, the management and iteration process of the stack can be optimized according to actual performance requirements to improve processing efficiency. Finally, when the stack is empty, the generated complete nested structure serialization result can be stored in the file system, database, or transmitted to other systems via the network. This serialization result not only saves storage space, but also facilitates subsequent processing and analysis. In this way, complex nested structure log data can be efficiently processed and serialized, providing strong technical support for complex business scenarios such as e-commerce platforms.

[0070] This embodiment utilizes a depth-first traversal rule starting from the root node, combined with a dynamic stack adjustment mechanism, to flexibly handle nested structures of varying depth and complexity, avoiding potential issues such as stack overflow or space waste caused by a fixed stack size. This mechanism not only enhances flexibility and adaptability, but also ensures that data at each level is correctly parsed and serialized through iterative processing, ultimately generating a complete nested structure serialization result.

[0071] In a further embodiment, when a hierarchical start feature is detected, dynamically adjusting the stack space to push the parsed hierarchical context information into the stack includes the following steps:

[0072] Step S5321: extracting the substructure length of the current nested level in the log data based on the virtual address;

[0073] The substructure length refers to the total byte length of all sublevels and their contents under the current level. In some embodiments, the substructure length exists in the form of a specific field in the header information of the log data or the metadata of the current level. For example, the metadata of each nested level may contain a field that clearly indicates the total length of all substructures under that level. The value of this field is usually an integer, which represents the total number of bytes of all substructures under the current level. For example, assuming that there is a field named "sub_structure_length" in the metadata of the current level, and its value is 1024 bytes, 1024 bytes will be used as the substructure length of the current level. By extracting the substructure length of the current nested level based on the virtual address, key data support is provided for dynamically adjusting the stack space.

[0074] Step S5322: Calculate the stack frame size required for the current nested level context information based on the substructure length; if the remaining stack space capacity is less than the required stack frame size, dynamically expand the stack space based on the stack frame size;

[0075] The stack frame size required for the context information of the current nested level is calculated based on the substructure length determined in the previous step. In some embodiments, the calculation of the stack frame size needs to take into account multiple factors, including the context information size and substructure length of the current nested level. The total size of the context information plus the substructure length constitutes the stack frame size required for the current nested level. Specifically, assuming that the context information size of the current nested level is 64 bytes and the substructure length is 1024 bytes, that is, the stack frame size required for the current nested level is 1088 bytes, the calculation process is dynamic, and the required stack frame size is calculated in real time based on the actual substructure length.

[0076] After calculating the stack frame size, the remaining capacity of the current stack space is checked. This refers to the amount of memory currently available for the stack. When the remaining capacity is less than the required stack frame size, the dynamic stack space expansion mechanism is triggered. This dynamic stack space expansion process is automated, allocating additional memory space based on the required stack frame size and linking it to the end of the current stack. For example, if the current remaining stack space capacity is 512 bytes and the calculated stack frame size is 1088 bytes, the stack space is dynamically expanded to allocate at least 1088 bytes of memory space. Dynamic expansion includes memory allocation operations, such as calling the operating system's memory allocation interface to obtain additional memory space.

[0077] In one embodiment, when dynamically expanding the stack space, the efficiency and security of memory allocation are considered. For example, a larger memory space is pre-allocated as the initial capacity of the stack, and gradually expanded when needed, rather than allocating just enough memory each time. This can reduce the number of memory allocation operations, improve overall performance, and update the management information of the stack space after the stack space is expanded, such as the stack top pointer and the total capacity of the stack space. This step can flexibly handle nested structures of different depths and complexities, avoiding parsing failures due to insufficient stack space. At the same time, the dynamic expansion mechanism can also optimize memory usage efficiency, ensuring that memory resources can be efficiently managed when processing large-scale nested structures.

[0078] Step S5323: Encapsulate the context information of the current nesting level into a stack frame, and push the stack frame into the dynamically adjusted stack.

[0079] After dynamically expanding the stack space, the context information of the current nested level is encapsulated into a stack frame, and the stack frame is pushed into the stack. Encapsulating a stack frame refers to organizing the context information of the nested level into a structured data unit. The format of this data unit is usually predefined to ensure that it can be correctly processed during push and subsequent pop operations. For example, a stack frame includes a level identifier, an event type, an event description, the number of sub-events, and the length of the substructure, among others. The level identifier refers to a unique number that identifies the current nested level, which is used to distinguish different levels during the parsing process; the event type is a description of the event type recorded at the current level, such as "order created", "payment completed", etc.; the event description is a detailed description of the current event, including specific event parameters or status information; the number of sub-events refers to the number of sub-events contained in the current level, which is used to control the depth of recursion during the parsing process; the parent level pointer is a pointer to the parent level of the current level, which is used to correctly return to the previous level when backtracking.

[0080] After encapsulation is complete, the stack frame is pushed onto the stack. Pushing adds a new stack frame to the top of the stack, making it the currently active frame. The last-in, first-out nature of the stack ensures that nested levels are processed in the correct order during subsequent parsing. After the push operation is complete, parsing of sub-events at the current level continues. During parsing, if a new nested level's starting feature is encountered, the above process is repeated, encapsulating a new stack frame and pushing it onto the stack. If the sub-event at the current level is parsed, the stack frame at the current level is popped from the stack, backtracking to the previous level and continuing to parse the remainder of the previous level. This approach enables efficient processing of complex nested log data, ensuring that data at each level is correctly parsed and managed. Ultimately, when all nested levels are parsed and the stack is empty, the generated serialized result will be a complete and accurate representation of the nested structure, suitable for subsequent data analysis, monitoring, or storage.

[0081] This embodiment dynamically extracts substructure lengths and adjusts stack space accordingly, enabling flexible handling of nested structures of varying depth and complexity. This avoids parsing failures due to insufficient stack space and optimizes memory usage. This embodiment not only improves adaptability and stability but also ensures data integrity and accuracy through structured stack frame encapsulation.

[0082] In a further embodiment, the steps are as follows:

[0083] Step S7100: Call the window sliding mechanism to calculate the average stack depth within a preset time period;

[0084] This step dynamically collects and analyzes the changes in stack depth through a time window of fixed length. Specifically, first set the length of a time window, which can be adjusted according to the actual application scenario and performance requirements. For example, when processing high-frequency log data, the time window can be set shorter to capture changes in stack depth more promptly; when processing low-frequency log data, the time window can be appropriately extended. The setting of the time window is the basis of the window sliding mechanism. In this step, the depth of the stack is continuously monitored, and the value of the stack depth is recorded within each time window. These values reflect the usage of the stack space within the time window. The window sliding mechanism can achieve continuous and dynamic acquisition of the latest stack depth information, rather than relying solely on one-time measurements. The average value of the stack depth within a preset time period is calculated. The average value is obtained by adding all the stack depth values in the window and then dividing it by the number of records in the window.

[0085] Step S7200: When it is detected that the average stack depth is lower than a preset multiple of the current stack capacity threshold for consecutive preset periods, a Gaussian distribution model is constructed based on the historical peak stack depth, and a lower limit of the confidence interval is calculated as the target shrinkage capacity;

[0086] When it is detected that the average stack depth is lower than a preset multiple of the current stack capacity threshold for several consecutive preset time periods, it means that the currently allocated stack space may be too large and has a certain degree of redundancy. In order to optimize memory usage efficiency, the stack space needs to be appropriately shrunk. To ensure that the shrunk stack space can still meet subsequent peak demands, a Gaussian distribution model is constructed based on the historical peak stack depth, and the target shrinkage capacity is determined by calculating the lower limit of the confidence interval. Among them, the preset multiple is a value less than 1, such as 0.7, which means that when the average stack depth is lower than 70% of the current stack capacity threshold, it is determined that the stack space may be redundant. If this condition is met for multiple consecutive time periods, the stack space shrinkage mechanism is triggered.

[0087] The historical peak stack depth refers to the highest stack depth reached over a period of time. This historical peak data is collected and a Gaussian distribution model is constructed based on this data. The Gaussian distribution model is a commonly used probability distribution model. Based on this Gaussian distribution model, the lower limit of the confidence interval is calculated. The confidence interval is a statistical concept that indicates the probability that the true value of the data falls within a certain interval, given a certain confidence level. For example, a 95% confidence interval indicates a 95% probability that the true value of the data will fall within this interval. An appropriate confidence level is selected, such as 95% or 99%, and the corresponding lower limit of the confidence interval is calculated based on the Gaussian distribution model. This lower limit ensures that in most cases, the reduced stack space will still meet actual needs. Finally, this lower limit of the confidence interval is used as the target shrinkage capacity. The target shrinkage capacity refers to the desired size of the stack space to shrink to. In some embodiments, this target capacity can be fine-tuned based on additional factors, such as other resource usage and current load conditions. In general, the target shrinkage capacity calculated based on the historical peak stack depth and the Gaussian distribution model provides a scientific and reasonable reference, allowing stack shrinkage to effectively reduce memory redundancy without causing subsequent stack overflows due to excessive shrinkage.

[0088] This dynamic adjustment mechanism based on statistical models and historical data can more intelligently manage stack space and improve memory usage efficiency.

[0089] Step S7300: Release redundant stack space based on the target shrinkage capacity.

[0090] After determining the size of the redundant stack space based on the target shrinkage capacity calculated in the previous step, a memory release operation is performed, including reallocation and adjustment of the stack space, to release the redundant and underutilized parts of the stack space to optimize memory usage efficiency. In one embodiment, the stack space is allocated as a continuous memory block, and the redundant memory is released by calling the memory management interface of the operating system. In the process of releasing memory, it is necessary to ensure that the existing data structure and context information in the stack are not destroyed. For example, assuming that the actual capacity of the current stack is 1MB and the target shrinkage capacity is 512KB, the redundant 512KB of memory is released. After the memory is released, the metadata of the stack is updated, including information such as the stack top pointer, the stack bottom pointer and the total capacity of the stack, to ensure the normal operation of the stack.

[0091] After freeing up redundant stack space, we continue to monitor stack usage. If, during subsequent log data parsing, the stack depth again approaches or exceeds the current stack capacity threshold, we reassess whether further stack space adjustments are necessary. This dynamic adjustment mechanism flexibly responds to log data fluctuations, avoiding the risk of stack overflows while ensuring efficient use of memory resources.

[0092] In this embodiment, by invoking a window sliding mechanism to calculate the average stack depth over a preset time period, stack usage can be dynamically monitored and analyzed, capturing changes in stack depth in real time. This allows for timely identification of redundant or insufficient stack space, providing accurate data support for subsequent stack space adjustments. Based on the calculated target shrinkage capacity, redundant stack space can be accurately released, optimizing memory usage efficiency while mitigating potential risks caused by insufficient stack space.

[0093] In a further embodiment, when the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is adopted to recursively parse and serialize the log data of the nested structure, and the following steps are included:

[0094] Step S8100: Map the disk file storing the serialized log data to the virtual address space, and generate the corresponding deserialized virtual address;

[0095] Map the disk file storing the serialized log data into the virtual address space and generate the corresponding deserial virtual address. Specifically, first, it is necessary to determine the location of the disk file storing the serialized log data. The target file can be located through the file path or file handle. After determining the location of the disk file, the memory mapping function is used to map the disk file into the virtual address space. Memory mapping is an efficient file access method that directly accesses the disk file without multiple disk read operations. During the memory mapping process, a virtual address range is created, which corresponds to the content of the disk file. The virtual address range is a continuous address area allocated in the virtual address space, and its size matches the size of the disk file. Through this embodiment, the generated deserial virtual address is the result of the memory mapping process, pointing to the starting position of the disk file content mapped in the virtual address space.

[0096] Step S8200: Based on the deserialized virtual address, read the header information of the corresponding log data to determine the organizational structure type of the log data;

[0097] Virtual addresses can be used to directly access data in disk files without complex address conversion or data copying. The header information of the corresponding log data is read based on the deserialized virtual address to determine the organizational structure type of the log data. Specifically, the starting position of the log data is first located by the deserialized virtual address. This address points to the starting point of the serialized log data, and the log data can be directly accessed and read. Since the data has been loaded into the virtual address space through the memory mapping mechanism, the memory mapping mechanism is used to avoid frequent disk I / O operations. Read the header information of the log data to parse the organizational structure type field to accurately identify the organizational structure type of the log data.

[0098] Step S8300: When the organizational structure of the event log stored in the log data is a process structure type, performing byte-level parsing on the serialized log data based on the deserialized virtual address, and restoring the parsed result to the original log data according to the preset serialization format;

[0099] On e-commerce platforms, log data is serialized into a byte stream for storage or transmission. When analyzing or processing serialized log data, the byte stream needs to be deserialized and restored to the original log data structure. For process-structured log data, the serialized log data is parsed at the byte level based on the deserialized virtual address, and the parsed results are restored to the original log data according to a preset serialization format. Specifically, the serialized log data is read byte by byte. Since process-structured log data is organized in a linear, continuous manner, each log record is independent and has a relatively fixed format. Each log record's fields are then parsed at the byte level. After parsing each field, the corresponding conversion and restoration operations are performed based on the field type. For example, for numeric fields such as timestamps and user IDs, they are directly converted from byte sequences to the corresponding numeric types. For example, the byte sequence representing the timestamp is converted to a long integer. For string-type fields such as operation types and operation objects, they are decoded from byte sequences to strings. Common decoding methods include UTF-8. For some special fields, such as date and time fields, further conversion is required based on their storage format (such as timestamp or specific date and time format) to restore them to a readable date and time format. For example, the timestamp must be converted to a specific year-month-day-hour-minute-second format.

[0100] In another embodiment, as the serialized data is parsed byte by byte and field restored, the structure of the original log data is gradually constructed. Each parsed log record is added to a log record list or array in the order in which it appears in the serialized data, ultimately forming a complete set of original log data. This step is continuous until the entire serialized log data is parsed and all log records are restored to their original structured data form. Implementing this step accurately restores the serialized log data stored in the disk file into original log data suitable for subsequent analysis and processing, enabling operations in various application scenarios such as user behavior analysis, system performance monitoring, and anomaly detection. Raw log data can be used in a variety of analysis scenarios, such as user behavior analysis, transaction anomaly detection, and system performance analysis. By analyzing user browsing paths, click behavior, purchase conversion rates, and other factors, user experience and recommendation algorithms can be optimized. By detecting abnormal behavior during transactions, such as payment failures and order anomalies, transaction security can be ensured. By analyzing system operation logs, performance bottlenecks can be detected and system performance optimized.

[0101] Step S8400: When the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is used to perform recursive byte-level parsing and deserialization processing on the nested structure of the serialized log data.

[0102] Before starting the deserialization process, it is first necessary to locate the starting position of the root node of the serialized log data through the deserialization virtual address, then initialize a stack, and encapsulate the context information of the root node into a stack frame and push it into the stack. In one embodiment, the serialized data is parsed layer by layer according to the depth-first traversal rule. During the parsing process, the start and end identifiers of the nested level are identified according to the preset serialization format identifier. When the start identifier of the nested level is detected, the stack space is automatically adjusted, the context information of the current level is encapsulated into a new stack frame, and it is pushed into the stack.

[0103] When parsing each nested level, the data of the current level is deserialized according to the preset serialization format. Deserialization refers to the conversion of a continuous byte stream into a structured data object for subsequent processing and analysis. When the end identifier of the nested level is parsed, the context information of the current level is popped from the stack. Due to the last-in-first-out feature of the stack, the context information popped out is the most recent one pushed onto the stack, that is, the context information of the nested level currently being processed. The parsed data is associated with its parent level based on the popped-out context information. For example, if the current level is a sub-event, the data of the sub-event is embedded in the corresponding position of the parent event based on the level identifier and data structure in the context information of the parent level. This step ensures that the logical relationship of the nested structure is correctly restored, so that the final deserialized data can accurately reflect the structure and content of the original log data.

[0104] During the deserialization process, stack usage is continuously monitored. If stack space is insufficient, it is dynamically expanded to ensure sufficient space for context information. This dynamic adjustment mechanism enables flexible handling of nested structures of varying depth and complexity, avoiding parsing failures due to insufficient stack space. Furthermore, based on the actual parsing progress and stack usage, no longer needed stack space is timely released to optimize memory efficiency. The entire deserialization process is iterative, continuously popping context information from the stack, parsing the data at the current level, and linking it to the parent level until the stack is empty, indicating that parsing and deserialization of all nested levels has been completed. Ultimately, the resulting reconstructed nested log data accurately reflects the structure and content of the original log data and can be used in a variety of analytical scenarios, such as user behavior analysis, transaction anomaly detection, and system performance analysis. This recursive parsing and deserialization approach efficiently handles complex nested log data, ensuring that data at each level is correctly parsed and restored, providing an accurate and efficient foundation for subsequent log data analysis and mining.

[0105] In this embodiment, by mapping the disk file storing serialized log data into the virtual address space and generating the corresponding deserialized virtual address, frequent disk I / O operations are avoided and the data reading efficiency is significantly improved. By utilizing the dynamic stack adjustment mechanism, the nested log data is recursively parsed and deserialized at the byte level, which can not only flexibly handle nested structures of different depths and complexities, but also avoid stack overflow or memory waste by dynamically adjusting the stack space.

[0106] In a further embodiment, when the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is used to perform recursive byte-level parsing and deserialization processing on the nested structure of the serialized log data, including:

[0107] Step S8310: Locate the starting position of the root node of the serialized log data based on the deserialized virtual address, and initialize a stack for storing context information of the deserialized log data level;

[0108] The starting position of the serialized log data is located by deserializing the virtual address. Since the log data is stored as a byte stream after serialization, the starting position of its root node is clearly identified. For example, in the serialized format, the root node begins with a specific marker or pattern. By parsing the serialized data pointed to by the deserialized virtual address, the starting marker of the root node is identified, thereby determining the exact location of the root node. After determining the starting position of the root node, a stack is initialized to store the context information of each layer during the deserialization process.

[0109] Step S8320: Parse the serialized data layer by layer according to the preset serialization format. When a nested level start identifier is detected, dynamically adjust the stack space to push the deserialization context information of the current level into the stack.

[0110] Starting from the root node, the serialized data is parsed layer by layer according to the preset serialization format. During the parsing process, the serialized data is read byte by byte, and the data structure and content of each layer are gradually parsed out. When the starting identifier of the nested level is detected, the stack space is dynamically adjusted to store the deserialization context information of the current level. In one embodiment, the required stack frame size is calculated based on the context information size of the current level and the estimated number of sub-levels. If the remaining capacity of the current stack space is not enough to accommodate the new stack frame, the stack space is automatically expanded. The context information of the current level is then pushed onto the stack, and the sub-events of the current level continue to be parsed. When a new nested level starting identifier is encountered, the above process is repeated, the stack space is dynamically adjusted, and the new context information is pushed onto the stack. This recursive parsing and stack space management mechanism can efficiently handle complex nested structures and ensure that data at each level can be correctly parsed and restored.

[0111] Step S8330: When the nested level end identifier is parsed, the context information of the current level is popped from the stack, and the parsed data is associated with the parent level according to the popped context information;

[0112] During the parsing of serialized data, the nested level end identifier in the serialized data is continuously monitored. When the end identifier is detected, it is recognized that the parsing of the current level has been completed. At this time, the context information of the current level is popped from the stack, and the parsing result of the current level is correctly associated with its parent level. When associating the parsing result of the current level with the parent level, the data of the child event is embedded in the corresponding position of the parent event according to the level identifier and data structure in the context information of the parent level, ensuring that the logical relationship of the nested structure is correctly restored, so that the final deserialized data can accurately reflect the structure and content of the original log data.

[0113] Step S8340: iteratively execute the above parsing and deserialization process until the stack is empty and all serialized data parsing is completed, generating reconstructed nested structure log data.

[0114] The parsing and deserialization process is repeated until the stack is empty, indicating that parsing and deserialization for all nested levels are complete. This step allows for the layer-by-layer parsing and restoration of nested log data, ensuring that each level of data is correctly processed. The nested relationships and logical structure of the original data are maintained. Once the stack is empty and all serialized data is parsed, the resulting reconstructed nested log data accurately reflects the structure and content of the original log data. This reconstructed log data can be used in a variety of analysis scenarios.

[0115] In this embodiment, by dynamically adjusting the stack space, the stack capacity can be flexibly expanded or shrunk according to actual needs, avoiding parsing failures due to insufficient stack space, and also reducing memory waste. In addition, the iterative execution mechanism can parse and restore nested log data layer by layer, ensuring that data at each level can be processed correctly and that the nested relationship and logical structure of the original data are maintained. This embodiment not only improves the efficiency of deserialization, but also enhances stability and reliability, providing strong technical support for complex business scenarios such as e-commerce platforms. When processing large-scale, complex log data, its performance and scalability advantages are obvious.

[0116] See also Figure 2 , a log data serialization device provided to meet one of the purposes of the present application is a functional embodiment of the log data serialization method of the present application. On the other hand, the device is a log data serialization device provided to meet one of the purposes of the present application, including a structure type determination module 5100, a process structure processing module 5200 and a nested structure processing module 5300, wherein the structure type determination module 5100 is used to locate the header information of the log data based on the virtual address in the virtual address space corresponding to the physical storage area of the log data, and parse the header information to determine the organizational structure type of the log data; the process structure processing module 5200 is used to perform byte-level parsing on the log data based on the virtual address when the organizational structure of the event log stored in the log data is a process structure type, and serialize the result obtained by the byte-level parsing according to a preset serialization format; the nested structure processing module 5300 is used to adopt a dynamic stack adjustment mechanism to recursively byte-level parse and serialize the log data of the nested structure when the organizational structure is a nested structure type.

[0117] In a further embodiment, the structure type determination module 5100 includes: a type field writing submodule, which is used to encode the organizational structure type of the log data into metadata in a preset format and write it into the type identification field of the log data header information; a virtual address generation submodule, which is used to map the starting address and length of the physical storage area where the log data is located to the virtual address space based on the page table mapping relationship of the memory management unit, and generate a corresponding virtual address to support parsing the header information of the corresponding log data according to the virtual address to determine the structure type of the log data.

[0118] In a further embodiment, the nested structure processing module 5300 includes: a stack initialization submodule, which is used to locate the root node of the nested structure based on the virtual address, and initialize the stack for storing the context information of the parsed nested level; a stack dynamic adjustment submodule, which is used to parse the nested structure at the byte level layer by layer according to the depth-first traversal rule, and when the level start feature is detected, dynamically adjust the stack space to push the parsed nested level context information into the stack; an information pop-up processing submodule, which is used to pop the corresponding nested level context information from the stack and serialize it according to a preset serialization format when the preset backtracking condition is met; a serialization result generation module, which is used to iteratively execute the above-mentioned parsing and serialization process until the stack is empty, and generate a complete nested structure serialization result.

[0119] In a further embodiment, the stack dynamic adjustment submodule includes: a substructure length determination submodule, which is used to extract the substructure length of the current nested level in the log data based on the virtual address; a stack frame size determination submodule, which is used to calculate the stack frame size required for the current nested level context information according to the substructure length. If the remaining capacity of the stack space is less than the required stack frame size, the stack space is dynamically expanded based on the stack frame size; and a stack frame push submodule, which is used to encapsulate the context information of the current nested level into a stack frame and push the stack frame into the dynamically adjusted stack.

[0120] In a further embodiment, it includes: a data acquisition submodule, which is used to call the window sliding mechanism to calculate the average stack depth within a preset time period; an adjustment parameter determination submodule, which is used to construct a Gaussian distribution model based on the historical peak stack depth when it is detected that the average stack depth is lower than a preset multiple of the current stack capacity threshold for a consecutive preset period, and calculate the lower limit value of the confidence interval as the target shrinkage capacity; a dynamic balance submodule, which is used to release redundant stack space based on the target shrinkage capacity.

[0121] In a further embodiment, the nested structure processing module 5300 includes: a virtual address mapping submodule, which is used to map the disk file storing serialized log data to the virtual address space to generate a corresponding deserial virtual address; a header information reading submodule, which is used to read the header information of the corresponding log data based on the deserial virtual address to determine the organizational structure type of the log data; a log data restoration submodule, which is used to perform byte-level parsing on the serialized log data based on the deserial virtual address when the organizational structure of the event log stored in the log data is a process structure type, and restore the parsing result to the original log data according to the preset serialization format; a deserialization submodule, which is used to adopt a dynamic stack adjustment mechanism to recursively byte-level parse and deserialize the nested structure of the serialized log data when the organizational structure is a nested structure type.

[0122] In a further embodiment, the deserialization submodule includes: a root node positioning submodule, which is used to locate the starting position of the root node of the serialized log data based on the deserialized virtual address, and initialize the stack for storing the context information of the deserialized log data level; a hierarchical parsing submodule, which is used to parse the serialized data layer by layer according to the preset serialization format, and when a nested level start identifier is detected, dynamically adjust the stack space to push the deserialization context information of the current level into the stack; a hierarchical end processing submodule, which is used to pop the context information of the current level from the stack when a nested level end identifier is parsed, and associate the parsed data to the parent level according to the popped context information; an iterative deserialization submodule, which is used to iteratively execute the above-mentioned parsing and deserialization process until the stack is empty and all serialized data are parsed, thereby generating reconstructed nested structure log data.

[0123] In order to solve the above technical problems, the embodiment of the present application also provides a computer device. Figure 3 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions, and the database may store a control information sequence, and when the computer-readable instructions are executed by the processor, the processor may implement a log data serialization method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor may execute the log data serialization method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 3The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0124] In this embodiment, the processor is used to execute Figure 2 The memory stores the program code and various data required to execute the modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the log data serialization device of this application, and the server can call the server's program code and data to execute the functions of all submodules.

[0125] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the log data serialization method of any embodiment of the present application.

[0126] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0127] Those skilled in the art will understand that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be interchanged, changed, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be interchanged, changed, rearranged, decomposed, combined, or deleted. Furthermore, the steps, measures, and schemes in the various operations, methods, and processes in the prior art that are open source and disclosed in this application may also be interchanged, changed, rearranged, decomposed, combined, or deleted.

[0128] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A log data serialization method, characterized in that: include: Locating header information of the log data based on a virtual address in a virtual address space corresponding to a physical storage area of the log data, and parsing the header information to determine an organizational structure type of the log data; When the organizational structure of the event log stored in the log data is a process structure type, performing byte-level parsing on the log data based on the virtual address, and serializing the result obtained by the byte-level parsing according to a preset serialization format; When the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is adopted to perform recursive byte-level parsing and serialization processing on the log data of the nested structure.

2. The log data serialization method according to claim 1, characterized in that: Locating header information of the log data based on a virtual address in a virtual address space corresponding to a physical storage area of the log data, and parsing the header information to determine the organizational structure type of the log data, including: Encode the organizational structure type of the log data into metadata in a preset format and write it into the type identification field of the log data header information; Based on the page table mapping relationship of the memory management unit, the starting address and length of the physical storage area where the log data is located are mapped to the virtual address space, and the corresponding virtual address is generated to support the parsing of the header information of the corresponding log data according to the virtual address to determine the structure type of the log data.

3. The log data serialization method according to claim 1, characterized in that: When the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is used to perform recursive byte-level parsing and serialization processing on the log data of the nested structure, including: Locating a root node of the nested structure based on the virtual address, and initializing a stack for storing context information for parsing the nested level; Parsing the nested structure layer by layer at the byte level according to a depth-first traversal rule, and when a level start feature is detected, dynamically adjusting the stack space to push the parsed nested level context information into the stack; When the preset backtracking condition is met, the corresponding nested level context information is popped from the stack and serialized according to the preset serialization format; The above parsing and serialization process is iteratively performed until the stack is empty, generating a complete nested structure serialization result.

4. The log data serialization method according to claim 3, characterized in that: When a hierarchical start feature is detected, the stack space is dynamically adjusted to push the parsed hierarchical context information into the stack, including: Extracting the substructure length of the current nested level in the log data based on the virtual address; Calculating the stack frame size required for the current nested level context information according to the substructure length, and dynamically expanding the stack space based on the stack frame size if the remaining capacity of the stack space is less than the required stack frame size; Encapsulate the context information of the current nesting level into a stack frame, and push the stack frame into the dynamically adjusted stack.

5. The log data serialization method according to any one of claims 1 to 4, characterized in that: include: Call the window sliding mechanism to calculate the average stack depth within the preset time period; When it is detected that the average stack depth is lower than a preset multiple of the current stack capacity threshold for consecutive preset periods, a Gaussian distribution model is constructed based on the historical peak stack depth, and a lower limit of the confidence interval is calculated as the target shrinkage capacity; Redundant stack space is released based on the target shrink capacity.

6. The log data serialization method according to any one of claims 1 to 4, characterized in that: When the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is used to perform recursive byte-level parsing and serialization processing on the log data of the nested structure, including: Map the disk file storing serialized log data to the virtual address space and generate the corresponding deserialized virtual address; Based on the deserialized virtual address, read the header information of the corresponding log data to determine the organizational structure type of the log data; When the organizational structure of the event log stored in the log data is a process structure type, performing byte-level parsing on the serialized log data based on the deserialized virtual address, and restoring the parsed result to the original log data according to the preset serialization format; When the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is adopted to perform recursive byte-level parsing and deserialization processing on the nested structure of the serialized log data.

7. The log data serialization method according to claim 6, characterized in that: When the organizational structure is a nested structure type, a dynamic stack adjustment mechanism is used to perform recursive byte-level parsing and deserialization processing on the nested structure of the serialized log data, including: Locating a root node start position of the serialized log data based on the deserialized virtual address, and initializing a stack for storing context information of the deserialized log data level; Parsing the serialized data layer by layer according to the preset serialization format, and when a nested level start identifier is detected, dynamically adjusting the stack space to push the deserialization context information of the current level into the stack; When the end identifier of the nested level is parsed, the context information of the current level is popped from the stack, and the parsed data is associated with the parent level based on the popped context information; The above parsing and deserialization process is iteratively executed until the stack is empty and all serialized data parsing is completed, thereby generating reconstructed nested structure log data.

8. A log data serialization device, characterized in that: include: a structure type determination module, configured to locate header information of the log data based on a virtual address in a virtual address space corresponding to a physical storage area of the log data, and parse the header information to determine an organizational structure type of the log data; a process structure processing module, configured to, when the organizational structure of the event log stored in the log data is a process structure type, perform byte-level parsing on the log data based on the virtual address, and serialize the result obtained by the byte-level parsing according to a preset serialization format; The nested structure processing module is used to adopt a dynamic stack adjustment mechanism to perform recursive byte-level parsing and serialization processing on the log data of the nested structure when the organizational structure is a nested structure type.

9. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images