File loading method, device, electronic device and storage medium

Abstract

The present invention relates to the field of computer technology, and provides a file loading method, device, electronic device, and storage medium, wherein the method comprises: based on the data structure of the file to be loaded, dividing the file to be loaded into multiple physical blocks, and parsing each physical block in parallel to extract relevant information of each record in each physical block; based on the numbering information and time information of each record, classifying and sorting each record to obtain multiple time trees; based on the time information of each record, determining the global time sequence of each record, and applying the global time sequence to divide any time tree into multiple logical blocks; based on the numbering information and position information of each record in each logical block, determining the mapping relationship between each record in each logical block and each record in each physical block, and applying the mapping relationship to load each logical block in parallel. The present invention can improve the parallelism of file loading, significantly shorten the file loading time, and improve loading efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a file loading method, device, electronic device and storage medium. Background Art

[0002] As software systems become increasingly complex, the demand for performance analysis is also growing. Trace Event Format (TEF), a universal time series event tracing format, uses JSON files as the primary carrier of performance data. Their ability to record event timelines, durations, and dependencies in detail makes them indispensable tools for performance tuning. These files typically contain tens of thousands of records, detailing the system's operation at different stages. By opening these files with appropriate visualization tools, developers can visually visualize the system's performance across various tasks, identifying potential performance issues and optimizing them.

[0003] However, loading and processing these large TEF JSON files is often limited by the complex dependencies between records, especially those with start and end points. The sequential nature of these events makes it difficult to effectively process the records in the JSON files in parallel, forcing them to be processed record by record. Even with sufficient hardware resources, the inability to fully utilize the advantages of parallel computing can still result in slow loading and parsing speeds, severely impacting the efficiency of performance analysis. Summary of the Invention

[0004] The present invention provides a file loading method, device, electronic device and storage medium, which are used to solve the defects of low file loading parallelism and slow loading speed in related technologies.

[0005] The present invention provides a file loading method, comprising:

[0006] Based on the data structure of the file to be loaded, the file to be loaded is divided into multiple physical blocks, and each physical block is parsed in parallel to extract relevant information of each record in each physical block, the relevant information including time information, number information and location information;

[0007] Classifying and sorting the records based on the number information and time information of each record to obtain multiple time trees;

[0008] Determine a global time sequence of the records based on the time information of the records, and apply the global time sequence to divide any time tree into a plurality of logical blocks;

[0009] Based on the number information and position information of each record in each logical block, a mapping relationship between each record in each logical block and each record in each physical block is determined, and the mapping relationship is applied to load the logical blocks in parallel.

[0010] According to a file loading method provided by the present invention, the numbering information of any record includes a block number, a record number and a thread number, the block number refers to the number of the physical block to which the any record belongs, the record number refers to the number of the any record in the physical block, and the thread number refers to the number of the thread corresponding to the any record.

[0011] According to a file loading method provided by the present invention, the records are classified and sorted based on the number information and time information of each record to obtain multiple time trees, including:

[0012] Classifying each record based on the event type to which each record belongs and the thread number of each record to obtain multiple record sets;

[0013] Based on the time information of each record in any record set, each record in the any record set is sorted to obtain a time tree corresponding to the any record set.

[0014] According to a file loading method provided by the present invention, each record in any logical block is divided into a main record and a pre-fetch record. Based on the number information and position information of each record in each logical block, a mapping relationship between each record in the logical block and each record in the physical block is determined, and the mapping relationship is applied to load the logical blocks in parallel, including:

[0015] Determine, based on the block number, record number, and location information of the main record in any logical block, a mapping relationship between the main record and each record in each physical block, and read the content of the main record from the physical block corresponding to the mapping relationship;

[0016] Based on the time information of each record in any one of the logical blocks, the content of the main record and the content of the pre-fetched record are sorted and integrated to obtain a record sequence corresponding to any one of the logical blocks. The content of the pre-fetched record is read from each physical block in advance based on the block number, record number and position information of the pre-fetched record in any one of the logical blocks.

[0017] According to a file loading method provided by the present invention, the step of determining the main record and the pre-fetched record in any logic block includes:

[0018] Determining a primary physical block corresponding to any one of the logical blocks from the physical blocks based on the block numbers of each record in the any one of the logical blocks;

[0019] Determine each record in the any logical block whose block number is the same as the number of the primary physical block as the primary record of the any logical block;

[0020] Each record in the any logical block whose block number is different from the number of the main physical block is determined as a pre-fetch record of the any logical block.

[0021] According to a file loading method provided by the present invention, the event type to which each record belongs includes a first type and a second type, and the event type of each record in each logic block is the first type;

[0022] The method further comprises:

[0023] Based on the multiple time trees, loading processing is performed on each record in each physical block whose event type is the second type.

[0024] According to a file loading method provided by the present invention, the loading process is performed on each record of the second event type in each physical block based on the multiple time trees, including:

[0025] Determine any record of the second type of event type in any physical block as a second type event, and obtain endpoint information of the second type event, the endpoint information including time information, a record number, and a thread number of each endpoint, and each endpoint including a start point and an end point;

[0026] Matching the thread numbers corresponding to the endpoints of the second type of event with the thread numbers corresponding to the multiple time trees to determine the time trees corresponding to the endpoints of the second type of event;

[0027] Based on the time information and record number of each endpoint of the second type event and the time tree corresponding to each endpoint, the connection data of the second type event is determined.

[0028] According to a file loading method provided by the present invention, determining the connection data of the second type event based on the time information and record number of each endpoint of the second type event and the time tree corresponding to each endpoint includes:

[0029] determining the horizontal coordinates of both ends of the connecting line based on the time information of each endpoint of the second type event;

[0030] Based on the time information and record number of each endpoint of the second type event, determine the level of each endpoint in the time tree corresponding to each endpoint, and determine the vertical coordinates of the two ends of the connection line by applying the level of each endpoint and the time tree corresponding to each endpoint;

[0031] Based on the horizontal coordinates and vertical coordinates at both ends of the connection line, connection data of the second type of event is obtained.

[0032] The present invention also provides a file loading device, comprising:

[0033] a physical block division unit, configured to divide the file to be loaded into a plurality of physical blocks based on the data structure of the file to be loaded, and to parse each physical block in parallel to extract relevant information of each record in each physical block, the relevant information including time information, number information, and location information;

[0034] A time tree construction unit, configured to classify and sort the records based on the number information and time information of each record to obtain a plurality of time trees;

[0035] a logic block division unit, configured to determine a global time sequence of the records based on the time information of the records, and to divide any time tree into a plurality of logic blocks by applying the global time sequence;

[0036] The parallel loading unit is used to determine the mapping relationship between each record in each logical block and each record in each physical block based on the number information and position information of each record in each logical block, and apply the mapping relationship to parallel load the logical blocks.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-described file loading methods is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned file loading methods when executed by a processor.

[0039] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned file loading methods.

[0040] The file loading method, device, electronic device and storage medium provided by the present invention can fully utilize the parallel processing capability of multi-core processors, reduce overall loading time and improve resource utilization by dividing the file to be loaded into multiple physical blocks and performing parallel parsing on each physical block. At the same time, relevant information of each record in each physical block can be extracted, including time information, numbering information and position information, etc., thereby pre-extracting the global time order so that each record in each time tree can be logically divided into blocks based on the global time order and a logical block loading plan can be formulated. By classifying and sorting each record based on the numbering information and time information of each record, multiple time trees can be constructed, thereby adjusting the order of each record to be arranged in chronological order, so that it can be divided into multiple logical blocks according to the global time order. According to the numbering information and position information of each record in each logical block, the mapping relationship between each record in each logical block and each record in each physical block can be determined. By applying the mapping relationship, the parallel loading of each logical block can be realized, thereby improving the parallelism of file loading, greatly shortening the loading time and improving loading efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 It is a flowchart of the file loading method provided by the present invention;

[0043] Figure 2 It is a structural diagram of the main records and pre-fetched records in the logic block provided by the present invention;

[0044] Figure 3 It is a structural diagram of the flow event connection provided by the present invention;

[0045] Figure 4 It is a structural schematic diagram of the file loading device provided by the present invention;

[0046] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0048] In the field of data processing and analysis, loading and parsing large JSON files is a critical step in scenarios such as performance analysis, system monitoring, and application optimization. These files, particularly those that conform to TEF, are widely used to record application or system performance data, including but not limited to the time and duration of events, as well as the dependencies between events. These files store rich performance data in a structured format, which is crucial for gaining a deeper understanding and optimizing system behavior. For example, some browsers store collected performance data as TEF JSON files. Opening this JSON file with the browser's built-in Trace Viewer (a visualization tool for viewing and analyzing trace data) allows for visual analysis, greatly facilitating performance tuning.

[0049] TEF's JSON files contain two main types of events. One is a completion event, which represents a sequential event with a clear start and end point that occurs on a logical timeline. Duration events, on the other hand, consist of two records: a start and an end point. These are often merged into a single event to simplify analysis. The processing of these merged events is similar to that of completion events. The other type is a flow event, which describes the flow relationship between completion events. A flow event indicates that one end of a completion event points to the end of another completion event. In the TEF JSON file, all completion events on each thread occur on a logical timeline. This is visualized as a function call graph, progressing from left to right and upside down from top to bottom. Flow events are visualized as a Bezier curve connecting the start and end points. It should be noted that a completion event represents the complete process of a logical activity from start to finish. These activities (i.e., completion events) are executed on specific threads. Therefore, each completion event is tagged or associated with the thread in which it occurred.

[0050] When using TEF JSON files for performance analysis, these JSON files can be loaded and parsed using visual performance analysis tools. File loading refers to reading file data stored on external storage devices (such as hard drives or network storage) into the computer's memory or processor so that the analysis tool can access and process the data. Common performance analysis tools primarily consist of a collection module and an analysis module. The collection module is responsible for efficiently recording various performance events into the TEF JSON file, focusing on two types of events: completion events, which represent a period of time, and flow events, which represent the flow of messages between threads. To reduce collection overhead and latency, the collection module saves the collected time series information to the file as quickly as possible. The generated completion event sequence does not guarantee temporal ordering, but instead offloads time-related processing tasks to the analysis module. The analysis module faces two major challenges when loading large TEF JSON files (e.g., 2-10GB): first, extracting the parent-child call relationships of the corresponding completion event sequence based on temporal nesting; second, quickly and efficiently rendering large amounts of data. Since the collected data may be out of order, the analysis module must first resolve the disorder problem to ensure that the events are arranged in chronological order and the processing is completed quickly within the limited memory peak limit.

[0051] However, when the analysis module loads and processes these large TEF JSON files, due to the dependencies and disorder between records of specific events such as the complete event, it is unable to effectively process different types of records or records of the same type but belonging to different event sequences in parallel. It can only process records one by one sequentially. This processing method cannot fully utilize the advantages of multi-core processors and is slow when loading large files. In addition, to address the disorder problem, a queue mechanism is usually used to temporarily store and sort records. However, since the time of subsequent records is difficult to predict, the temporal relationship of records can only be processed by relying on local time windows. The lack of global time processing capabilities makes it impossible to completely resolve the disorder of complete events. As a result, errors or omissions may occur when constructing the event timeline, affecting the accuracy and reliability of performance analysis.

[0052] To this end, the present invention provides a file loading method, which obtains a global time sequence by pre-extracting the time information of the records at one time, and sorts the records according to the time information. In this way, the file can be divided into multiple logical blocks, and the processing of each logical block can be independent and parallel. This not only solves the problem of event disorder, but also makes full use of multi-core processors to make the loading process highly parallel, thereby achieving the purpose of greatly improving file loading efficiency, thereby overcoming the above-mentioned defects.

[0053] It should be noted that the file loading method provided by the present invention can be applied to loading JSON files of large TEFs, and can also be applied to other scenarios where a large number of out-of-order records are loaded in an orderly manner. The present invention does not specifically limit this. The following will take loading a JSON file of TEF as an example to introduce the technical solution of the present invention.

[0054] Figure 1 It is a flowchart of the file loading method provided by the present invention, such as Figure 1 As shown, the method includes:

[0055] Step 110: based on the data structure of the file to be loaded, the file to be loaded is divided into multiple physical blocks, and each physical block is parsed in parallel to extract relevant information of each record in each physical block, the relevant information including time information, number information, and location information;

[0056] It should be noted that a file to be loaded refers to a file that needs to be processed, read, or parsed to obtain information. In embodiments of the present invention, the file to be loaded may be a data file containing a large number of records, which may be stored in a specific format (such as JSON or XML). For example, the file to be loaded may be a TEF JSON file. The data structure of the file to be loaded refers to the organization and representation of the data in the file, including the arrangement of records, the type and order of fields within the records, and possible delimiters or markers.

[0057] Specifically, based on the data structure of the file to be loaded, the file can be segmented to divide it into multiple file blocks of similar size. The file blocks at this stage can be referred to as physical blocks to distinguish them from the logical blocks described later. Here, a physical block refers to the independent part of the file formed after segmentation, and each physical block contains a portion of the file's data records. The physical blocks are arranged in the order in which they appear in the file to be loaded. Each physical block has a unique number and location information for the physical block in the file to be loaded. The above unique number can be referred to as the block number.

[0058] For example, for a TEF JSON file, the file can be divided into multiple physical blocks of similar size based on each node in the file. Here, a node refers to a lightweight data structure in memory of a section of content contained in characters [] or {} that reflects the hierarchical relationship in the JSON file. This block division is implemented based on the data structure of the JSON file, and each node will only appear completely in one physical block. Subsequently, JSON parallel parsing technology can be used to parse these physical blocks in parallel to extract relevant information such as time information, number information, and location information of all records in each physical block. Here, JSON parallel parsing technology is a processing framework that can quickly parse and process large JSON files in parallel at low memory peak. Through this technology, each physical block can be parsed in parallel on different threads or processes to extract relevant information of each record in each physical block.

[0059] Specifically, when using JSON parallel parsing technology to parse each physical block in parallel, multiple threads or processes can be used to process the physical blocks in parallel, with each thread or process processing a physical block and independently parsing the data within that physical block. Here, the parsing process specifically involves reading the data within the physical block and converting it into a data structure (such as nested objects and arrays) that the program can understand and manipulate based on JSON syntax rules. During or after parsing, the constructed data structure can be traversed to extract the required information. For example, recursion or other traversal algorithms can be used to access each object and array in the data structure and extract relevant information, such as field values and timestamps, to obtain relevant information about each record in the physical block. It should be understood that parsing is a prerequisite for information extraction. The raw data in the JSON file must first be converted into a data structure that the program can understand, and then traversing this structure and extracting the required information can be performed. Therefore, these two steps, parsing and extraction, together constitute the process of extracting the required information from the JSON file.

[0060] It is understandable that in the TEF JSON file, a record refers to a single performance event. Each record contains detailed information about a specific operation or event, such as the timestamp of the event, the duration of the event, the thread number associated with the event, and other metadata related to the event. Each record in a physical block refers to the records divided into each physical block. The relevant information of each record refers to the data extracted from each record that is useful for subsequent processing. For example, this relevant information may include time information, number information, location information, etc., where time information refers to the timestamp and / or duration of the record; number information refers to the identifier or serial number of the record, which is used to uniquely identify the record or track the relationship between records; location information refers to the position of the record in the original file (i.e., the file to be loaded) (such as byte offset, byte length, etc.), which helps to quickly locate the original data when needed.

[0061] Step 120: classify and sort the records based on their number information and time information to obtain multiple time trees.

[0062] Specifically, after extracting the relevant information for all records in each physical block, each record can be classified based on its numbering information, and all records can be divided into different record sets. For example, the numbering information of each record can include a thread number. When classifying the records, records with the same thread number can be grouped into a record set, where each record set represents a series of time-series events recorded on a timeline. Here, the thread number refers to the number of the thread associated with the record. In a multi-threaded environment, each thread has a unique number or identifier that is used to distinguish different threads.

[0063] It's important to note that in TEF JSON files, the order in which each record appears is called the file order, while the order in which the events corresponding to each record actually occurred is called the chronological order. The time information in each record implicitly reflects the nested hierarchical relationship between records (i.e., the chronological order). Due to the out-of-order nature of complete events, the file order and chronological order don't completely correspond. Records with earlier timestamps may appear later in the file. Therefore, it's necessary to adjust the order of each record from file order to chronological order to facilitate faster processing during visualization.

[0064] In order to arrange the records in chronological order, after classifying the records and obtaining multiple record sets, for each record set, the nesting level of each record can be determined according to the time information of each record in the record set, and all records can be sorted according to these time information and nesting levels, thereby forming a series of attributes arranged along the time axis with the root node at the top and the leaf nodes at the bottom, which is called a time tree. Here, the time tree refers to a data structure used to represent a series of events or records on a timeline. The time tree can be regarded as a tree structure in which each node represents an event or record. It should be understood that the time tree is a data structure formed by re-sorting the time information of all records on a thread, together with the numbering information of each record, according to the hierarchical nested inclusion relationship of time. The order in this structure is the time order, which is also the display order in the timeline view.

[0065] Step 130: determining a global time sequence of the records based on the time information of the records, and applying the global time sequence to divide any time tree into a plurality of logical blocks;

[0066] Specifically, after extracting the relevant information of all records in each physical block, the global time order can be determined based on the time information of each record. Specifically, all records from different physical blocks can be collected into a unified data structure, such as a list or database table; then the time information of all records can be traversed to determine the hierarchical nested inclusion relationship of each record time, and the earliest and latest timestamps can be found to obtain the global time order, which will serve as the basis for segmenting the time tree. Here, the global time order refers to the time order of all records within the global time range. This order is global and takes into account all records, not just the records within a certain time tree.

[0067] After determining the global time order, the records in each time tree can be divided into equal sizes according to the global time order to obtain multiple logical blocks. Here, a logical block is a logical grouping, which refers to a subset of records cut from all records in a single time tree. Before splitting, the size of the logical block can be determined based on the number of all records in a single time tree. For each time tree, it can be split according to the size of the logical block.

[0068] For example, when any time tree is divided into logical blocks, all records in the time tree can be divided according to the arrangement order and nested inclusion relationship of all records in the time tree in the global time sequence. It should be understood that each record in each time tree has been arranged in chronological order. Therefore, for each time tree, these records can be allocated to the corresponding logical blocks in sequence by traversing each record in the time tree. For example, the first 1000 records can be divided into the first logical block, the next 1000 records can be divided into the second logical block, and so on. Each record will be divided into a logical block.

[0069] It is understandable that logical block segmentation is for a single time tree, that is, all records on the time tree corresponding to a single thread are segmented from front to back in chronological order to obtain multiple logical blocks. A physical block corresponds to a section of content in the file to be loaded, which may include records from different threads, while a logical block only contains records from a certain thread. The logical block segmentation processing of different time trees is independent and parallel. For the same time tree, the loading processing of multiple logical blocks of the time tree is also independent and parallel.

[0070] Step 140 : Based on the number information and position information of each record in each logical block, determine the mapping relationship between each record in the logical block and each record in the physical block, and apply the mapping relationship to load the logical blocks in parallel.

[0071] It should be noted that the logical block usually only contains some relevant information of the record (such as time information, number information, location information, etc.), and does not contain the detailed content of the record. This allows the data to be organized and indexed logically (i.e., in global time order) to more efficiently process and analyze data, while reducing memory usage and increasing processing speed. The detailed content of the record is still stored in the physical block, which is the actual location where the data is stored. Therefore, when loading the logical block, the corresponding record content will be read from the physical block based on the mapping relationship between each record in the logical block and each record in the physical block, so as to be used for subsequent performance visualization analysis. It should be understood that the above-mentioned step 140 is intended to perform parallel loading processing on multiple logical blocks within a single time tree, and the parallel loading processing process of the logical blocks in other time trees is carried out in accordance with step 140.

[0072] Specifically, within any logical block, each record has corresponding numbering and location information. Based on this information, a corresponding record can be uniquely located within the physical block. Therefore, for each logical block, a mapping table can be constructed based on the numbering and location information of all records within the logical block. This mapping table records the correspondence between each record in the logical block and each record in the physical block. Once the mapping table is constructed, JSON parallel parsing technology can be used to scan it again, reading the physical blocks corresponding to each logical block in batches according to the order of each logical block, and then parallel processing can begin. Specifically, based on the requirements for parallel loading, logical blocks can be assigned to different processing units (such as threads, processes, or computer nodes) for parallel loading. Each processing unit, based on the assigned logical block and the mapping table, directly reads record data from the corresponding physical block. Finally, the read record data is sorted according to the chronological order of the records within the logical block to obtain the corresponding record sequence. It should be understood that the record sequence obtained after parallel loading is a dataset organized according to the logical blocks and loaded into memory or a database. These datasets can be encoded and packaged as needed and sent downstream for further analysis, processing, or display.

[0073] The method provided by the embodiment of the present invention divides the file to be loaded into multiple physical blocks and parses each physical block in parallel, which can not only fully utilize the parallel processing capabilities of the multi-core processor, reduce the overall loading time, and improve resource utilization, but also extract the relevant information of each record in each physical block, including time information, number information, and location information, etc., thereby pre-extracting the global time sequence so that each record in each time tree can be logically divided into blocks based on the global time sequence and a logical block loading plan can be formulated. By classifying and sorting each record based on the number information and time information of each record, multiple time trees can be constructed, thereby adjusting the order of each record to be arranged in time sequence, so that it can be divided according to the global time sequence to obtain multiple logical blocks. According to the number information and location information of each record in each logical block, the mapping relationship between each record in each logical block and each record in each physical block can be determined. By applying this mapping relationship, the parallel loading of each logical block can be realized, thereby improving the parallelism of file loading, greatly shortening the loading time, and improving loading efficiency.

[0074] Based on the above embodiment, the numbering information of any record includes a block number, a record number and a thread number, wherein the block number refers to the number of the physical block to which the any record belongs, the record number refers to the number of the any record in the physical block, and the thread number refers to the number of the thread corresponding to the any record.

[0075] Specifically, for each physical block, when extracting relevant information for each record in the physical block, the extracted information may include the record's time information, number information, and location information. The number information includes the block number, record number, and thread number. Here, the block number refers to the number of the physical block to which the record belongs. When the file to be loaded is divided into multiple physical blocks, each physical block is arranged in the order in which it appears in the file to be loaded, and each physical block has a unique number, which is the block number. The block number allows for quick location of the physical block where the record is located, improving data access efficiency.

[0076] A record number is a number or sequence number assigned to a record in a physical block. Within a physical block, each record has a unique number that distinguishes different records within the same physical block. After locating the physical block containing a record using the block number, the record number and location information can be used to further locate a specific record within that physical block, allowing for subsequent access to the record's detailed contents.

[0077] The thread number refers to the number of the thread associated with the record. In the TEF JSON file, the time information of all records associated with each thread is on a logical timeline. The thread number can be used to classify all records in the file, so that the records can be further sorted later.

[0078] Based on any of the above embodiments, the event type to which each record belongs includes a first type and a second type, and the event type of each record in each logical block is the first type.

[0079] It's important to note that TEF's JSON files contain two main types of events: complete events, which record the start time and duration of an event; and flow events, which represent the relationship between two complete events. Therefore, the event types of each record in the JSON file can be divided into type 1 and type 2, where type 1 is complete and type 2 is flow.

[0080] Specifically, because the loading process for complete events and flow events is different, the event type of each record in the file to be loaded can be identified to identify the records corresponding to complete events and flow events, respectively, so that the loading process for complete events and flow events is independent of each other. It should be understood that in the TEF JSON file, there may not be event types directly named complete and flow. To identify complete events and flow events, this can be achieved by analyzing the relevant fields of each record in the JSON file (such as the name field, timestamp, thread number, etc.) and analyzing the chronological order of the records.

[0081] Based on any of the above embodiments, step 120 specifically includes:

[0082] Step 121: classify each record based on the event type to which each record belongs and the thread number of each record to obtain multiple record sets;

[0083] Step 122: sort the records in any record set based on the time information of each record in the record set to obtain a time tree corresponding to the record set.

[0084] Specifically, when classifying each record, the focus is mainly on the records corresponding to the complete event. Therefore, for all records whose event type is identified as a complete event, these records can be classified according to the thread number of each record, and records with the same thread number can be divided into the same record set. For example, for all records whose event type is a complete event, all records with a thread number of 1 can be divided into the first record set, all records with a thread number of 2 can be divided into the second record set, and so on. It should be understood that a record set refers to a set of records with the same characteristics (i.e., the same thread number) formed in the above classification process, and each record set represents a series of time-series events recorded on a logical timeline.

[0085] After completing the record classification and obtaining multiple record sets, all records in each record set can be further sorted so that each record is arranged in chronological order, thereby constructing a time tree, which facilitates the subsequent logical block segmentation of the time tree. Specifically, for each record set, various sorting algorithms (such as quick sort, merge sort, heap sort, etc.) can be used to sort the records according to the time information of all records in the record set. After the sorting is completed, the records will be arranged in chronological order. Then, the nesting level of each record can be determined based on the time information of each record. According to the arrangement order and nesting level of each record, these records can be organized into a tree structure arranged along the time axis with the root node at the top and the leaf nodes at the bottom, which is called a time tree. Each node in the time tree represents a record.

[0086] Based on any of the above embodiments, each record in any logic block is divided into a main record and a pre-fetched record, and the step of determining the main record and the pre-fetched record in any logic block includes:

[0087] Determining a primary physical block corresponding to any one of the logical blocks from the physical blocks based on the block numbers of each record in the any one of the logical blocks;

[0088] Determine each record in the any logical block whose block number is the same as the number of the primary physical block as the primary record of the any logical block;

[0089] Each record in the any logical block whose block number is different from the number of the main physical block is determined as a pre-fetch record of the any logical block.

[0090] It should be noted that, considering that most records in the file to be loaded are ordered, in order to improve file loading efficiency, the records in the logical blocks can be divided into two types: main records and prefetched records. The main records corresponding to all logical blocks are read in batches and can be shared by all logical blocks; the prefetched records are read on a fine-grained, on-demand basis, that is, read separately for each logical block. Before loading each logical block, the prefetcher can prefetch the prefetched records of the logical block in advance. Therefore, during the loading process, only the main records of the logical block need to be read, thereby improving loading efficiency.

[0091] Specifically, primary records refer to those records in a logical block whose block number is the same as the number of the primary physical block corresponding to the logical block. Prefetch records refer to all other records in the logical block except the primary record, that is, those records whose block number is different from the primary physical block number. These records can be pre-loaded or retained in memory through prefetching technology to improve the efficiency of subsequent access. Prefetching is an optimization technology that aims to reduce the delay caused by waiting for data to be loaded from a slow storage device (such as a hard disk) to a fast storage device (such as memory). It should be understood that for a certain time tree, when the logical block is split, the prefetch record list for each logical block can be determined. By pre-reading according to this list in advance by the prefetch thread, the processing requirements of the logical block can be met.

[0092] Figure 2 This is a schematic diagram of the structure of the main records and pre-fetched records in the logic block provided by the present invention, such as Figure 2 As shown in the figure, after any time tree is divided into multiple logical blocks of equal size according to the global time sequence, the records in each logical block are scattered in several physical blocks, and one of these physical blocks can be selected as the primary physical block. For example, the physical block corresponding to the most records can be selected as the primary physical block corresponding to the logical block. Specifically, after the time tree is divided, the record range assigned to a certain logical block is as follows: Figure 2 As shown, the logical block includes 8 records, among which the block number of the record corresponding to the first orange block on the upper left is physical block 0, the block number of the record corresponding to the two orange blocks below the orange block is physical block 1, the block number of the record corresponding to the three green blocks is physical block 2, and the block number of the record corresponding to the two orange blocks on the right is physical block 3. Obviously, the number of records corresponding to physical block 2 is the largest, so physical block 2 can be determined as the primary physical block of the logical block. After determining the primary physical block, all records falling within the primary physical block can be used as the main records of the logical block (such as Figure 2 The green square shown in Figure 2), other records that fall outside the primary physical block are used as prefetch records (such as Figure 2 shown as orange squares).

[0093] In an embodiment of the present invention, by dividing the time tree into multiple logical blocks according to the global time sequence, and dividing the records of each logical block into main records and pre-fetched records, not only can the front-end and back-end dependencies of each logical block be eliminated, making the subsequent processing of each logical block independent of each other, but also the logical blocks divided by time sequence natively eliminate the disorder problem of complete events, laying the foundation for the correctness and high performance of subsequent parallel block processing.

[0094] Based on any of the above embodiments, step 140 specifically includes:

[0095] Step 141: Determine a mapping relationship between the primary record and each record in each physical block based on the block number, record number, and location information of the primary record in any logical block, and read the content of the primary record from the physical block corresponding to the mapping relationship.

[0096] Step 142, based on the time information of each record in any logical block, sort and integrate the content of the main record and the content of the pre-fetched record to obtain a record sequence corresponding to any logical block, and the content of the pre-fetched record is pre-read from each physical block based on the block number, record number and position information of the pre-fetched record in any logical block.

[0097] Specifically, after dividing into multiple logical blocks, these logical blocks can be scanned using JSON parallel parsing technology in order, and the main physical blocks corresponding to the logical blocks can be read in batches to start parallel loading processing. Specifically, before loading each logical block, a prefetcher can be used to prefetch the prefetch records of the logical block in advance. During the loading process, after reading the records contained in the main physical block corresponding to the logical block (i.e., the content of the main record), the content of the main record can be integrated with the prefetched records extracted in advance. It should be understood that in order to read the content of the main record of each logical block, the corresponding record in the physical block can be located according to the block number, record number and location information of the main record, so that the content of these records can be read from the physical block.

[0098] It is understood that obtaining the contents of the prefetch record for each logical block can be achieved through a prefetcher. The prefetcher reads the contents of each prefetch record one by one directly from the file to be loaded, based on the block number, record number, and location information of each prefetch record, and in accordance with the prefetch record list provided by the logical block, thereby preparing the required prefetch records for each logical block in advance.

[0099] For each logical block, after obtaining the contents of the block's primary records, the primary records can be sorted according to their chronological order within the logical block when integrating them with the pre-fetched records, resulting in a record sequence that corresponds one-to-one with the corresponding record range in the time tree. This record sequence can then be encoded and packaged and sent downstream, serving as input to subsequent processing steps for further data analysis, report generation, and more.

[0100] Based on any of the above embodiments, the method further includes:

[0101] Step 150 : Based on the multiple time trees, load each record in each physical block whose event type is the second type.

[0102] Specifically, when classifying and sorting each record and constructing a time tree, the records processed are mainly records with the event type of complete event. Therefore, the multiple logical blocks obtained by segmenting the time tree in the above embodiment are actually the various blocks of the complete event. By loading each logical block in parallel, the loading and processing of the complete event in the JSON file is realized. Since the flow event represents the association relationship between the two ends of the complete event, the loading and processing of the flow event can be realized according to the time tree, and can be carried out in parallel and independently with the processing of the complete event. It should be noted that the second type mentioned above is flow.

[0103] Based on any of the above embodiments, step 150 specifically includes:

[0104] Step 151: Determine any record in any physical block whose event type is the second type as a second type event, and obtain endpoint information of the second type event, wherein the endpoint information includes time information, a record number, and a thread number of each endpoint, and each endpoint includes a start point and an end point.

[0105] Specifically, the second type of event mentioned above is a flow event. When loading flow events from a TEF JSON file, you can first read all records with the event type "flow" in the physical block and parse their contents to extract endpoint information related to the flow event. Here, the endpoint information of a flow event refers to the key data points that describe the start and end of the flow event. This information can include the specific time information (such as a timestamp) of each endpoint (i.e., the starting and ending points), the record number and thread number corresponding to each endpoint, and so on.

[0106] Step 152: Match the thread number corresponding to each endpoint of the second type of event with the thread numbers corresponding to the multiple time trees to determine the time trees corresponding to each endpoint of the second type of event.

[0107] Step 153 : determining the connection data of the second type event based on the time information and record number of each endpoint of the second type event and the time tree corresponding to each endpoint.

[0108] Specifically, after obtaining the endpoint information for all flow events, the start and end points of each flow event can be categorized by thread, allowing them to be sent to the corresponding time tree for binding calculation. The binding results are then combined and paired into a single connection, which can be sent downstream for further processing. Binding calculation here refers to determining which record in the time tree each endpoint is associated with and determining the hierarchy of this record.

[0109] It's understandable that a flow event represents the flow from one completion event to another. These two completion events can occur on the same thread or on different threads. Therefore, for each flow event, the thread numbers corresponding to its starting and ending points may be the same or different. Since time trees correspond one-to-one to threads, by matching the thread number corresponding to each endpoint of the flow event with the thread number corresponding to the time tree, each endpoint can be sent to the corresponding time tree for binding.

[0110] A flow event is represented as a line from its starting point to its ending point. When loading a flow event through a time tree, the horizontal and vertical coordinates of the line's ends can be determined in the time tree based on the flow event's starting and ending time information, the record number, and the corresponding time trees. This provides the flow event's line data. Here, the flow event's line data refers to the result of further processing based on the flow event's endpoint information. It describes the flow event's continuity and possible interactions from start to finish on the timeline.

[0111] It can be understood that the connection data of the flow event is the data structure that represents the flow direction of the flow event, which may include the identifiers of the starting point and the end point (that is, the unique identifier of the complete event), the time information of the starting point and the end point, the hierarchical information of the starting point and the end point in the time tree, etc.

[0112] Based on any of the above embodiments, step 153 specifically includes:

[0113] Step 1531: Determine the horizontal coordinates of the two ends of the connecting line based on the time information of each endpoint of the second type event;

[0114] Step 1532: Based on the time information and record number of each endpoint of the second type event, determine the level of each endpoint in the time tree corresponding to each endpoint, and use the level of each endpoint and the time tree corresponding to each endpoint to determine the vertical coordinates of the two ends of the connection line;

[0115] Step 1533: Obtain the connection data of the second type of event based on the horizontal coordinates and vertical coordinates at both ends of the connection.

[0116] Specifically, a flow event is represented by a line pointing from the starting point to the end point. When the flow event is loaded through the time tree, the start time and end time of the flow event can be determined based on the time information of the start and end points of the flow event (such as timestamps). Based on the start time and end time, the horizontal coordinates of the two ends of the line can be determined in the time tree. Based on the time information and record number of the starting and end points, the records corresponding to the starting and end points can be located in the time tree. Based on the level of the record, the hierarchical information of the starting and end points in the time tree can be determined. Based on the time tree where the starting and end points are located and the corresponding hierarchical information, the vertical coordinates of the two ends of the line can be determined. Based on the horizontal and vertical coordinates of the two ends of the line, the connection data of the flow event can be obtained. This connection data can be encoded, packaged and sent to the downstream for further analysis and visualization.

[0117] Figure 3 This is a schematic diagram of the flow event connection structure provided by the present invention, such as Figure 3 As shown in the figure, the curved lines with arrows are the lines corresponding to flow events. In time tree 1, record A corresponds to level 0, record B corresponds to level 1, and record C corresponds to level 2. Similarly, in time tree 2, record D corresponds to level 0, record E corresponds to level 1, and record F corresponds to level 2. The following describes the specific loading process of flow events using the flow event line marked as ①.

[0118] First, the starting and ending points of a flow event are categorized by thread number. For the flow event marked as ①, based on the thread numbers of its starting and ending points, we can determine that the starting point corresponds to time tree 1 and the ending point corresponds to time tree 2, respectively. Based on the time information of the starting and ending points, we can determine the abscissas of the two ends of the line on the horizontal time axis. Subsequently, in time tree 1, based on the time information and record number of the starting point, we can determine that the record corresponding to the starting point is record B (i.e., level 1); in time tree 2, based on the time information and record number of the ending point, we can determine that the record corresponding to the ending point is record F (i.e., level 2). On the vertical axis, the starting point's ordinate is determined by the time tree identifier of the starting point and the starting point's level, while the ending point's ordinate is determined by the time tree identifier of the ending point and the ending point's level. Finally, by combining and matching the abscissas and ordinates of the two ends of the line, we can construct the line corresponding to the flow event.

[0119] Based on any of the above embodiments, an embodiment of the present invention provides a highly parallelized method for loading a JSON file, the method comprising:

[0120] Step S1, physical chunking: Using JSON parallel parsing technology, the TEF JSON file is divided into chunks of similar size based on the nodes in the file. This chunking is based on the JSON file structure; a node appears entirely within only one chunk. Chunks are arranged in the order they appear in the file. Each chunk has a unique number (i.e., chunk number) and file location information. To distinguish it from the logical chunks described later, the chunking at this stage is referred to as physical chunking. This chunking step forms the foundation for subsequent steps. Chunking serves as the execution granularity for each thread task, and should be as uniform as possible.

[0121] Step S2, extracting time: Using JSON parallel parsing technology, quickly scan each physical block obtained in step S1, extracting the time information, thread number, block number, and record number within each record in the physical block, and temporarily storing the location information of each record in the file. Extracting time is a prerequisite for achieving global awareness, and temporarily storing the location information of each record in the file allows for subsequent pre-fetching of records.

[0122] Step S3, constructing a time tree: all records are divided into multiple tracks (i.e., the record set in the above embodiment) according to the event type and thread number. Each track represents a series of time series events recorded on this timeline, and one track corresponds to a time series data table. All records in each track are sorted according to time and nested level to form a tree structure with the root node at the top and the leaf nodes at the bottom arranged along the time axis, which is called a time tree. The time tree corresponds one-to-one to the data displayed on the final timeline, and also corresponds to the order of records in the corresponding time series data table. This order is called the time order, and the order of appearance of records in the original JSON file is called the file order. The time tree retains the mapping correspondence between the time order of records and the file order. This step generates the time order of records and maintains the mapping correspondence between the time order and the file order.

[0123] Step S4, logical block partitioning: Divide any time tree into multiple logical blocks of equal size according to the global chronological order of the records. The records in each logical block are dispersed across several physical blocks. The physical block with the largest number of corresponding records is selected as the primary physical block for that logical block. The remaining records outside this primary physical block are prefetched records. The records in the logical block are then divided into two parts: primary records (i.e., all records within the primary physical block) and prefetched records. This step eliminates the dependencies between logical blocks, making subsequent processing independent of each other. Furthermore, the chronologically ordered logical blocks inherently eliminate the problem of out-of-order completion events, laying the foundation for the correctness and high performance of subsequent parallel block processing.

[0124] Step S5, process flow events: classify the start and end points of the flow event according to the track, and send them to the time tree corresponding to each track for binding calculation. Then, the binding results are merged and paired into a line, and sent to the downstream for further processing. The flow event represents the flow from one complete event to another complete event, and is ultimately expressed as a line with an arrow. According to the time information of the start and end points of the flow event, the start time and the end time can be determined. Based on this, the horizontal coordinates of the two ends of the line can be determined; according to the identification of the time tree itself and the level of the record found by time, the vertical coordinates of the two ends of the line can be determined. It should be understood that the flow event processing only depends on the time tree. Therefore, the flow event can be processed independently in parallel with the logical block of the complete event.

[0125] Step S6, parallel processing: Scan again based on JSON parallel parsing technology, read the physical blocks corresponding to the logical blocks in batches according to the logical block order in step S4, and start parallel processing. Specifically, for each logical block, a prefetcher can be used to prefetch the prefetch records of the logical block in advance, and integrate them with the records contained in the main physical block corresponding to the logical block, sort them according to the time sequence of each record in the logical block, adjust them into a record sequence that corresponds one-to-one to the corresponding record range in the time tree, and then encode and package them and send them downstream for further performance analysis and visual display. It should be understood that the prefetcher can use the record location information saved in step S2, and load each record directly from the JSON file according to the prefetch record list provided by the logical block, and then prepare the prefetch records required for each logical block in advance. This step is the core of parallel processing, and is performed simultaneously with step S5 to load and process complete events and flow events respectively.

[0126] In an embodiment of the present invention, by preprocessing the JSON file to be loaded, sorting the records in the file according to time information, and then dividing the data into logical blocks, the order of processing of each logical block is eliminated. At the same time, the processing of flow events and complete events is independent of each other, and the processing of each block of the complete event is independent of each other. This improves the parallelism of the block processing, fully utilizes multiple cores, and significantly shortens the loading time. In addition, by dividing the logical blocks based on global time order, it can be ensured that even the completion event at the end of the file with the time at the beginning can be correctly processed, thus completely solving the problem of out-of-order completion events.

[0127] Based on any of the above embodiments, Figure 4 This is a schematic diagram of the structure of the file loading device provided by the present invention. Figure 4 As shown, the device includes:

[0128] A physical block division unit 410 is configured to divide the file to be loaded into multiple physical blocks based on the data structure of the file to be loaded, and to perform parallel parsing on each physical block to extract relevant information of each record in each physical block, the relevant information including time information, number information, and location information;

[0129] A time tree construction unit 420 is configured to classify and sort the records based on the number information and time information of each record to obtain multiple time trees;

[0130] a logic block unit 430 for determining a global time sequence of the records based on the time information of the records, and dividing any time tree into a plurality of logic blocks by applying the global time sequence;

[0131] The parallel loading unit 440 is used to determine the mapping relationship between each record in each logical block and each record in each physical block based on the number information and position information of each record in each logical block, and apply the mapping relationship to parallel load the logical blocks.

[0132] The device provided by the embodiment of the present invention can fully utilize the parallel processing capabilities of multi-core processors, reduce overall loading time, and improve resource utilization by dividing the file to be loaded into multiple physical blocks and performing parallel parsing on each physical block. At the same time, it can also extract relevant information of each record in each physical block, including time information, number information, and location information, etc., thereby pre-extracting the global time order so that each record in each time tree can be logically divided into blocks based on the global time order and a logical block loading plan can be formulated. By classifying and sorting each record based on the number information and time information of each record, multiple time trees can be constructed, thereby adjusting the order of each record to be arranged in time order, so that it can be divided according to the global time order to obtain multiple logical blocks. According to the number information and location information of each record in each logical block, the mapping relationship between each record in each logical block and each record in each physical block can be determined. By applying this mapping relationship, the parallel loading of each logical block can be realized, thereby improving the parallelism of file loading, significantly shortening the loading time, and improving loading efficiency.

[0133] Based on any of the above embodiments, the numbering information of any record includes a block number, a record number and a thread number, wherein the block number refers to the number of the physical block to which any record belongs, the record number refers to the number of any record in the physical block, and the thread number refers to the number of the thread corresponding to any record.

[0134] Based on any of the above embodiments, the time tree construction unit 420 is specifically configured to:

[0135] Classifying each record based on the event type to which each record belongs and the thread number of each record to obtain multiple record sets;

[0136] Based on the time information of each record in any record set, each record in the any record set is sorted to obtain a time tree corresponding to the any record set.

[0137] Based on any of the above embodiments, each record in any logic block is divided into a main record and a pre-fetch record. Accordingly, the parallel loading unit 440 is specifically configured to:

[0138] Determine, based on the block number, record number, and location information of the main record in any logical block, a mapping relationship between the main record and each record in each physical block, and read the content of the main record from the physical block corresponding to the mapping relationship;

[0139] Based on the time information of each record in any one of the logical blocks, the content of the main record and the content of the pre-fetched record are sorted and integrated to obtain a record sequence corresponding to any one of the logical blocks. The content of the pre-fetched record is read from each physical block in advance based on the block number, record number and position information of the pre-fetched record in any one of the logical blocks.

[0140] Based on any of the above embodiments, the step of determining the main records and pre-fetched records in any of the logic blocks includes:

[0141] Determining a primary physical block corresponding to any one of the logical blocks from the physical blocks based on the block numbers of each record in the any one of the logical blocks;

[0142] Determine each record in the any logical block whose block number is the same as the number of the primary physical block as the primary record of the any logical block;

[0143] Each record in the any logical block whose block number is different from the number of the main physical block is determined as a pre-fetch record of the any logical block.

[0144] Based on any of the above embodiments, the event type to which each record belongs includes a first type and a second type, and the event type of each record in each logical block is the first type; accordingly, the device further includes an event processing unit, the event processing unit being configured to:

[0145] Based on the multiple time trees, loading processing is performed on each record in each physical block whose event type is the second type.

[0146] Based on any of the above embodiments, the event processing unit specifically includes:

[0147] an acquiring subunit, configured to determine any record of the second type of event in any physical block as a second type of event, and to acquire endpoint information of the second type of event, the endpoint information including time information, a record number, and a thread number of each endpoint, each endpoint including a start point and an end point;

[0148] a matching subunit, configured to match the thread number corresponding to each endpoint of the second type event with the thread numbers corresponding to the multiple time trees, and determine the time trees corresponding to each endpoint of the second type event;

[0149] The determination subunit is configured to determine the connection data of the second type event based on the time information of each endpoint of the second type event, the record number, and the time tree corresponding to each endpoint.

[0150] Based on any of the above embodiments, the determination subunit is specifically configured to:

[0151] determining the horizontal coordinates of both ends of the connecting line based on the time information of each endpoint of the second type event;

[0152] Based on the time information and record number of each endpoint of the second type event, determine the level of each endpoint in the time tree corresponding to each endpoint, and determine the vertical coordinates of the two ends of the connection line by applying the level of each endpoint and the time tree corresponding to each endpoint;

[0153] Based on the horizontal coordinates and vertical coordinates at both ends of the connection line, connection data of the second type of event is obtained.

[0154] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the file loading method, which includes: based on the data structure of the file to be loaded, dividing the file to be loaded into multiple physical blocks, and parsing each physical block in parallel to extract relevant information of each record in each physical block, wherein the relevant information includes time information, numbering information and location information; based on the numbering information and time information of each record, classifying and sorting the records to obtain multiple time trees; based on the time information of each record, determining the global time order of each record, and applying the global time order to divide any time tree into multiple logical blocks; based on the numbering information and location information of each record in each logical block, determining the mapping relationship between each record in each logical block and each record in each physical block, and applying the mapping relationship to load the logical blocks in parallel.

[0155] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the relevant art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0156] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the file loading method provided by the above methods, which method includes: based on the data structure of the file to be loaded, dividing the file to be loaded into multiple physical blocks, and parsing each physical block in parallel to extract relevant information of each record in each physical block, wherein the relevant information includes time information, numbering information and location information; based on the numbering information and time information of each record, classifying and sorting each record to obtain multiple time trees; based on the time information of each record, determining the global time order of each record, and applying the global time order to divide any time tree into multiple logical blocks; based on the numbering information and location information of each record in each logical block, determining the mapping relationship between each record in each logical block and each record in each physical block, and applying the mapping relationship to load each logical block in parallel.

[0157] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the file loading method provided by the above-mentioned methods, the method comprising: based on the data structure of the file to be loaded, dividing the file to be loaded into multiple physical blocks, and parsing each physical block in parallel, extracting relevant information of each record in each physical block, the relevant information including time information, numbering information and location information; based on the numbering information and time information of each record, classifying and sorting the records to obtain multiple time trees; based on the time information of each record, determining the global time order of each record, and applying the global time order to divide any time tree into multiple logical blocks; based on the numbering information and location information of each record in each logical block, determining the mapping relationship between each record in each logical block and each record in each physical block, and applying the mapping relationship to load the logical blocks in parallel.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0159] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A file loading method, characterized in that: include: Based on the data structure of the file to be loaded, the file to be loaded is divided into multiple physical blocks, and each physical block is parsed in parallel to extract relevant information of each record in each physical block, the relevant information including time information, number information and location information; Classifying and sorting the records based on their number information and time information to obtain multiple time trees, where the time tree is a data structure used to represent a series of records on a timeline; Determine a global time sequence of the records based on the time information of the records, and apply the global time sequence to divide any time tree into a plurality of logical blocks; Based on the number information and position information of each record in each logical block, a mapping relationship between each record in each logical block and each record in each physical block is determined, and the mapping relationship is applied to load the logical blocks in parallel.

2. The file loading method according to claim 1, characterized in that: The numbering information of any record includes the block number, record number and thread number. The block number refers to the number of the physical block to which the any record belongs, the record number refers to the number of the any record in the physical block, and the thread number refers to the number of the thread corresponding to the any record.

3. The file loading method according to claim 2, characterized in that: Based on the number information and time information of each record, the records are classified and sorted to obtain multiple time trees, including: Classifying each record based on the event type to which each record belongs and the thread number of each record to obtain multiple record sets; Based on the time information of each record in any record set, each record in the any record set is sorted to obtain a time tree corresponding to the any record set.

4. The file loading method according to claim 2, characterized in that: Each record in any logical block is divided into a main record and a pre-fetch record. Based on the number information and position information of each record in each logical block, a mapping relationship between each record in each logical block and each record in each physical block is determined, and the mapping relationship is applied to load each logical block in parallel, including: Determine, based on the block number, record number, and location information of the main record in any logical block, a mapping relationship between the main record and each record in each physical block, and read the content of the main record from the physical block corresponding to the mapping relationship; Based on the time information of each record in any one of the logical blocks, the content of the main record and the content of the pre-fetched record are sorted and integrated to obtain a record sequence corresponding to any one of the logical blocks. The content of the pre-fetched record is read from each physical block in advance based on the block number, record number and position information of the pre-fetched record in any one of the logical blocks.

5. The file loading method according to claim 4, characterized in that: The step of determining the main record and the pre-fetch record in any one of the logic blocks comprises: Determining a primary physical block corresponding to any one of the logical blocks from the physical blocks based on the block numbers of each record in the any one of the logical blocks; Determine each record in the any logical block whose block number is the same as the number of the primary physical block as the primary record of the any logical block; Each record in the any logical block whose block number is different from the number of the main physical block is determined as a pre-fetch record of the any logical block.

6. The file loading method according to any one of claims 1 to 5, characterized in that: The event types to which each record belongs include a first type and a second type, and the event type of each record in each logical block is the first type; The method further comprises: Based on the multiple time trees, loading processing is performed on each record in each physical block whose event type is the second type.

7. The file loading method according to claim 6, characterized in that: The loading process of each record of the second type of event type in each physical block based on the multiple time trees includes: Determine any record of the second type of event type in any physical block as a second type event, and obtain endpoint information of the second type event, the endpoint information including time information, a record number, and a thread number of each endpoint, and each endpoint including a start point and an end point; Matching the thread numbers corresponding to the endpoints of the second type of event with the thread numbers corresponding to the multiple time trees to determine the time trees corresponding to the endpoints of the second type of event; Based on the time information and record number of each endpoint of the second type event and the time tree corresponding to each endpoint, the connection data of the second type event is determined.

8. The file loading method according to claim 7, characterized in that: The determining of the connection data of the second type event based on the time information of each endpoint of the second type event, the record number, and the time tree corresponding to each endpoint includes: determining the horizontal coordinates of both ends of the connecting line based on the time information of each endpoint of the second type event; Based on the time information and record number of each endpoint of the second type event, determine the level of each endpoint in the time tree corresponding to each endpoint, and determine the vertical coordinates of the two ends of the connection line by applying the level of each endpoint and the time tree corresponding to each endpoint; Based on the horizontal coordinates and vertical coordinates at both ends of the connection line, connection data of the second type of event is obtained.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the file loading method according to any one of claims 1 to 8 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the file loading method according to any one of claims 1 to 8 is implemented.

Images