File analysis method and system and vehicle

Abstract

The invention is suitable for the technical field of file management, and provides a file analysis method and system and a vehicle, the file analysis method and system are applied to a file analysis system, the file analysis system runs at a browser end, and the file analysis system comprises the steps that binary header information of a to-be-analyzed file is analyzed according to the byte position, obtained in advance, of the binary header information in the to-be-analyzed file, and the binary header information of the to-be-analyzed file is obtained; binary index position information of the to-be-analyzed file is obtained; according to the binary index position information, reading and analyzing binary index data of the to-be-analyzed file to obtain index data, determining a target data block meeting a preset requirement according to basic information of each data block, and reading the target data block; and analyzing the data in the read target data block to obtain formatted target data. According to the invention, the problem that the cost of cloud data storage and data analysis is increased due to the fact that a large amount of file data is analyzed and stored at the cloud in the prior art can be solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of file management, and particularly relates to a file analysis method and system and a vehicle. BACKGROUND

[0002] In the research of algorithm optimization, verification and positioning problems in the field of automatic driving and robots, researchers need to repeatedly use a large amount of sensor data and system states (such as CPU load states) many times. In order to store and retrieve efficiently, these data are usually stored as files in the Cyber Record data format. In order to play back the files in the Cyber Record data format through visualization for researchers to use, the current mainstream solution is to do the visualization playback in the cloud.

[0003] The visualization playback in the cloud is to store a large number of Cyber Record files in the cloud, and then to pre-analyze the large number of Cyber Record files into structured format files, which are also stored in the cloud. When a user needs to use data within a certain range, the analyzed structured format files within the certain range are directly selected, and a visual interactive interface is used to display the analyzed structured format files to the user for playback, which is a web application.

[0004] The inventor found that storing a large number of Cyber Record files in the cloud and pre-analyzing the large number of Cyber Record files into structured format files and storing them in the cloud increases the cost of data storage in the cloud and the running cost of analyzing data. Moreover, for a web application, the analyzed large files may not be read normally, and abnormal data may be displayed to the user. SUMMARY

[0005] Embodiments of the application provide a file analysis method and system and a vehicle, which are applied in a browser end, and can solve the problem of increasing the cost of data storage in the cloud and the cost of analyzing data caused by pre-analyzing a large number of target files in the cloud in the prior art.

[0006] In a first aspect, embodiments of the application provide a file analysis method applied to a file analysis system, wherein the file analysis system runs in a browser end, and the file analysis method comprises the following steps. analyzing binary header information of the to-be-analyzed file according to the byte position of the binary header information in the to-be-analyzed file, to obtain binary index position information of the to-be-analyzed file; wherein the to-be-analyzed file is a Cyber Record file. According to the binary index position information, binary index data of the file to be parsed is read and parsed to obtain index data; wherein, the index data stores basic information of a plurality of data blocks; According to the basic information of each data block, a target data block meeting a preset requirement is determined and read; The data in the read target data block is parsed to obtain formatted target data.

[0007] According to the binary index position information, binary index data of the file to be parsed is read and parsed to obtain index data; wherein, the index data stores basic information of a plurality of data blocks; according to the basic information of each data block, a target data block meeting a preset requirement is determined and read; the data in the read target data block is parsed to obtain formatted target data. The file to be parsed is a Cyber Record file, so that only the target data block meeting the user's requirement is parsed, the entire Cyber Record file does not need to be parsed, the formatted target data is obtained, and the parsing can be directly implemented on the browser side without relying on the cloud to store the Cyber Record file in advance and without needing to store the parsed file on the cloud in advance, thereby saving the computing resources for parsing data on the cloud and the storage resources for storing the parsed file, and effectively solving the problem of increasing the cost of data storage and parsing on the cloud caused by storing a large amount of Cyber Record file data parsed in advance on the cloud.

[0008] In a possible implementation manner of the first aspect, the reading and parsing of the binary index data of the file to be parsed according to the binary index position information comprises: According to the binary index position information, the binary index data of the file to be parsed is read through a file system; The binary index data is deserialized and parsed through a parsing system to obtain index data.

[0009] Since the binary index position information is parsed, the binary index data of the file to be parsed can be read by the file system according to the binary index position information, and then the binary index data can be deserialized and parsed by the parsing system, so as to obtain the index data, thereby achieving the purpose of obtaining the index data by the parsing system and the file system, so that subsequent target data blocks meeting the requirements of a user can be determined according to the index data.

[0010] In a possible implementation of the first aspect, the basic information includes time information and binary position information. The method further includes: determining, according to the time information of each data block, at least one target data block meeting the preset requirement from the plurality of data blocks and a reading order of each target data block; reading, according to the binary position information of each target data block, data in each target data block by the file system in the reading order of each target data block.

[0011] The basic information of the data block includes time information of the data block and binary position information of the data block, at least one target data block meeting the preset requirement can be determined from the plurality of data blocks according to the time information of each data block, and a reading order of each target data block, for each target data block, the target data block can be found according to the binary position information of the target data block in the reading order of the target data block, data in the target data block can be read by the file system, and for all target data blocks, data in all target data blocks can be read in the reading order, since the reading order of the target data block is determined according to the time information, the data read in the reading order is approximately sorted according to the time information, and the time for sorting the structured target data in the future can be saved.

[0012] In a possible implementation of the first aspect, the time information includes a start time and an end time, and the start time and the end time are determined according to a timestamp of each data in the data block. The preset requirement includes a preset time range. The method further includes: For each data block, it is determined whether the time information of the data block coincides with the preset time range according to the start time, the end time, and the preset time range. When the time information of the data block overlaps with the preset time range, the data block is determined to be the target data block; The reading order of all the target data blocks is determined based on the start time of each target data block.

[0013] The time information includes start and end times, which are determined based on the timestamps of each data item in the data block. Preset requirements include a preset time range. For each data block, based on the start and end times and the preset time range, it is determined whether the data block's time information overlaps with the preset time range. When the data block's time information overlaps with the preset time range, the data block is identified as a target data block. This allows target data blocks that meet the preset requirements to be identified from the index data, enabling them to be read from the file system subsequently without needing to read data blocks that do not meet the preset requirements. Furthermore, the reading order of all target data blocks can be determined based on the start time of each target data block, ensuring that the read target data blocks are sorted according to their start time, which is beneficial for the data parsed later to be roughly sorted according to the time information.

[0014] In one possible implementation of the first aspect, the data carries a timestamp; The preset requirements include a preset time range; The step of parsing the data in the read target data block to obtain formatted target data includes: Based on the timestamps of the data in the target data block and the preset time range, the data in the target data block is filtered to obtain target data that meets the preset requirements; The target data is parsed by the parsing system to obtain the formatted target data.

[0015] The data in the data block carries a timestamp, and the preset requirements include a preset time range. Based on the timestamp and preset time range of the data in the target data block, the data in the target data block is filtered to obtain the target data that meets the preset requirements. The target data is then parsed by the parsing system to obtain formatted target data. By using the method of filtering first and then parsing, the target data can be determined first, and then only the target data needs to be parsed. It is not necessary to parse all the data in all the target data blocks, thereby saving the computing power and time costs of parsing.

[0016] In one possible implementation of the first aspect, the basic information includes pattern information corresponding to each data item; The target data is parsed by the parsing system to obtain the formatted target data, including: Based on the pattern information corresponding to each target data, determine the original definition of each target data; Based on the original definition of each target data, determine the target parsing method for each target data; According to the target parsing method for each target data, the parsing system performs deserialization parsing on each target data to obtain formatted target data.

[0017] The basic information of each data block includes the pattern information corresponding to the data in that data block. For any target data in each target data block that is read, the original definition of the target data can be determined according to the pattern information corresponding to the target data. Based on the original definition, the target parsing method of the target data can be determined. Through the parsing system, the target data can be deserialized and parsed according to the target parsing method. Dynamic decoding is achieved for all target data without the need to decode the entire file to be parsed in advance, so as to obtain the formatted target data that meets the user's requirements.

[0018] In one possible implementation of the first aspect, before parsing the binary header information of the file to be parsed, the method further includes: When the file to be parsed is a local file, the binary header information of the file to be parsed is read directly through the file system.

[0019] When the file to be parsed is a remote file, the binary header information of the file to be parsed is read through the Hypertext Transfer Protocol.

[0020] Before parsing the binary header information of a file, it is necessary to read the binary header information of the file through the file system. When the file to be parsed is a local file, the binary header information of the file to be parsed can be read directly through the file system. When the file to be parsed is a remote file, the binary header information of the file to be parsed can also be read directly through the Hypertext Transfer Protocol so that the parsing system can parse the binary header information.

[0021] In one possible implementation of the first aspect, when there are multiple formatted target data, each of the formatted target data carries a timestamp; After parsing the data in the read target data block to obtain formatted target data, the method further includes: Based on the timestamp, each of the formatted target data is sorted to obtain the sorted formatted target data.

[0022] When there are multiple formatted target data, each formatted target data carries a timestamp. Since the target data blocks are sorted data blocks, the read target data blocks are also in order. However, there may be two target data blocks where the target data is not in order. The data in the read target data blocks is parsed to obtain the formatted target data. After that, each formatted target data can be sorted according to the timestamp to obtain the sorted formatted target data.

[0023] Secondly, embodiments of this application provide a file parsing system that runs on a browser and includes a file system and a parsing system; wherein: The parsing system is used to parse the binary header information of the file to be parsed based on the byte position of the pre-acquired binary header information in the file to be parsed, so as to obtain the binary index position information of the file to be parsed; wherein, the file to be parsed is a Cyber ​​Record file; The file system is used to read the binary index data of the file to be parsed based on the binary index position information; The parsing system is also used to parse the binary index data of the file to be parsed to obtain index data; wherein, the index data stores basic information of multiple data blocks; The parsing system is also used to determine target data blocks that meet preset requirements based on the basic information of each data block; The file system is also used to read the target data block that meets the preset requirements; The parsing system is also used to parse the data in the read target data block to obtain formatted target data.

[0024] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the above-mentioned embodiments.

[0025] Fourthly, embodiments of this application provide a vehicle that includes the aforementioned file parsing system.

[0026] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0027] Sixthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute any of the methods described above.

[0028] The beneficial effects of this application embodiment compared with the prior art are as follows: By using a file parsing system running on the browser, the binary header information of the file to be parsed is parsed according to the byte position of the binary header information in the file to be parsed, and the binary index position information of the file to be parsed is obtained. Based on the binary index position information, the binary index data of the file to be parsed can be read and parsed to obtain the index data of the file to be parsed. Since the index data stores the basic information of multiple data blocks, the target data block that meets the preset requirements can be determined based on the basic information of each data block, and read to obtain the data in the target data block. Then, the data in the target data block is parsed to obtain the formatted target data, which is the data that meets the user's needs. The file to be parsed is a Cyber ​​Record file, so only the target data block that meets the user's needs can be parsed, without parsing the entire Cyber ​​Record file to obtain the formatted target data. Moreover, the parsing can be directly implemented on the browser side without relying on the pre-stored Cyber ​​Record in the cloud. This method eliminates the need to pre-parse the entire CyberRecord file in the cloud and then store the parsed file there. This saves computing resources for parsing data in the cloud and storage resources for storing the parsed file, effectively solving the problem of increased cloud data storage and parsing costs caused by pre-parsing and storing large amounts of CyberRecord file data in the cloud in existing technologies. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating a file parsing method provided in an embodiment of this application.

[0031] Figure 2 This is a schematic diagram of the structure of a cyber record file provided in an embodiment of this application.

[0032] Figure 3 This is a flowchart illustrating another file parsing method provided in another embodiment of this application.

[0033] Figure 4 This is a flowchart illustrating another file parsing method provided in another embodiment of this application.

[0034] Figure 5 This is a schematic diagram of the structure of a file parsing system provided in an embodiment of this application. Detailed Implementation

[0035] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0036] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0037] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0038] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0039] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0040] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0041] In the fields of autonomous driving and robotics, a large amount of sensor data and system status are recorded in Cyber ​​Record format. Cyber ​​Record data can be stored and retrieved efficiently. To efficiently visualize and replay the aforementioned sensor data and system status for use by researchers, visualizing and replaying Cyber ​​Record data in the cloud is currently the mainstream solution.

[0042] Currently, the most common visualization solution involves storing large amounts of Cyber ​​Record format data in the cloud, pre-parsed this Cyber ​​Record data, and then using DreamView to replay the parsed data to meet the user's needs. This parsing method mainly relies on a dedicated C++ toolchain, and to achieve local playback, it requires a cyber record environment.

[0043] Furthermore, Dreamweaver's web pages lack the ability to directly parse Cyber ​​Record format data. The Cyber ​​Record file needs to be pre-parsed completely in the cloud before the parsed data meeting the user's requirements is sent to the web page. This method of pre-parsing the entire Cyber ​​Record file in the cloud not only increases the cost of cloud data storage but also the computational cost of cloud data parsing. Moreover, if the parsed file is too large, the web page may not be able to read it correctly, potentially leading to errors. Therefore, providing a method for efficiently parsing Cyber ​​Record format files on the browser side is of great significance for reducing cloud computational and storage costs.

[0044] This application provides a file parsing method applied to a file parsing system, which runs on a browser. (See also...) Figure 1 , Figure 1This is a flowchart illustrating a file parsing method according to an embodiment of this application, including: Step S11: Based on the byte position of the binary header information in the file to be parsed obtained in advance, the binary header information of the file to be parsed is parsed to obtain the binary index position information of the file to be parsed.

[0045] Step S12: Based on the binary index position information, read and parse the binary index data of the file to be parsed to obtain the index data; wherein, the index data stores the basic information of multiple data blocks.

[0046] Step S13: Based on the basic information of each data block, determine the target data block that meets the preset requirements and read it.

[0047] Step S14: Parse the data in the read target data block to obtain formatted target data.

[0048] It should be noted that the file to be parsed in this embodiment is a Cyber ​​Record file. A Cyber ​​Record file is a file in the Cyber ​​Record format. The Cyber ​​Record format is a binary data recording and playback format used in real-time systems such as autonomous driving and robotics. It can store multi-source heterogeneous data generated during the operation of autonomous vehicles or robots in an efficient and structured manner. For example, the Cyber ​​Record format can be used to store sensor data (images, point clouds, radar, IMU, etc.), vehicle status (speed, steering angle, battery status, etc.), control commands (planned paths, decision results, etc.), and debugging information.

[0049] Specifically, the vehicle is equipped with a file parsing system that runs on the browser side, not in the cloud. This system parses Cyber ​​Record files to extract the data requested by the user and provides it to the user's applications. For example, the parsed sensor data can be used to train autonomous driving models or obstacle avoidance models.

[0050] like Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a cyber record file provided in an embodiment of this application, the file to be parsed ( Figure 2 The Cyber ​​Record File in the text is a Cyber ​​Record file, which includes a file header ( Figure 2 (Header in the middle), N data blocks ( Figure 2 Chunk 1...Chunk N) and an index data ( Figure 2The index is used in the parsing process. Here, N is an integer greater than 1, representing the number of data blocks in the file to be parsed. The entire file to be parsed stores serialized binary data.

[0051] The file header is the beginning of the file to be parsed, and it is also in binary form, including binary header information. This binary header information includes the start time, end time, number of channels (the number of data patterns), file size, and binary index position information, all represented in binary data. The binary index position information represents the position of the index in the file to be parsed, expressed in binary data. The start and end times of the file to be parsed represent the start and end times of data collection recorded in the file, respectively.

[0052] The number of channels refers to the number of recording channels in the file to be parsed, representing the amount of data pattern information. This data pattern information is determined by the original definition of the data, which in turn determines the type of the original data—that is, the type of original data corresponding to the data in the Cyber ​​Record file.

[0053] File size represents the size of the binary data in the file being parsed.

[0054] The data blocks of the file to be parsed record specific data messages, which are also in binary form.

[0055] The index of the file to be parsed records the basic information of each data block. The basic information of each data block includes the time information, location information and channel information of each data block. The time information is the time of the data in the data block; the location information is the location of the data block in the file to be parsed; the channel information represents the mode information of the data in the data block, including the protocol descriptor, which is used to deserialize the data in the data block.

[0056] The file parsing system consists of a file system and a parsing system. The file system is used to obtain the file to be parsed, and the parsing system is the parser used to parse the file to be parsed.

[0057] In an optional example, such as Figure 3 As shown, Figure 3This is a flowchart illustrating another file parsing method provided in another embodiment of this application. First, the file is input through a file input layer, where the file to be parsed is fed into the file parsing system. The file input layer is used to input files and can accept two data sources: local files (blobs) and remote URL files (browser-remote files). The file system includes a streaming reader, which can load data blocks from the file to be parsed in chunks, eliminating the need to download the entire file. This reduces memory usage and avoids system crashes caused by loading large amounts of data simultaneously.

[0058] The parsing system includes a core parsing layer, which parses the header of the file to be parsed. This layer extracts the file's metadata and time range, which can include start and end times. The metadata includes information about the file's basic attributes, structure, or content characteristics, and is typically stored in a specific area of ​​the file header (e.g., the file header itself).

[0059] The core parsing layer is also used to build indexes for data blocks, preload the built indexes, sort the indexes of the data blocks according to the start time of the data blocks, and cache the resulting index data. Figure 3 Index DB cache in the database.

[0060] The core parsing layer is also used to process data blocks and load target data blocks whose time range meets the user's requirements.

[0061] The parsing system also includes a message decoding layer, which is used to decode the data messages in the target data block, that is, to deserialize and parse the data to obtain the decoded data.

[0062] The parsing system also includes a message processing layer, which performs time-series management on the decoded data, sorts the decoded data according to its timestamps, and then outputs the visualization. The visualized output can be seamlessly integrated with existing visualization systems for display.

[0063] The file parsing system also includes an error recovery layer, which is used to identify errors in the file parsing process and to recover from or skip the identified errors.

[0064] In another optional example, such as Figure 4 As shown, Figure 4 This is a flowchart illustrating another file parsing method provided in another embodiment of this application. Before parsing the file to be parsed, initialization is required. Here, the file to be parsed can be a local binary file or a URL file (a remote file on the browser side). The purpose of initialization is to initialize the method for reading local or remote binary files, enabling the parsing system to read the binary data of the file to be parsed.

[0065] After initialization, the byte position of the binary header information in the file to be parsed is fixed; that is, starting from the 8th byte of the file header, the next 8 bytes constitute the binary header information. Figure 4 The file header (8n, 8n) is read from the file system to obtain the binary header information of the file to be parsed, and then the binary header information is sent to the parsing system. The binary header information is the header information represented in binary data form.

[0066] Since the binary header information includes binary index location information, after the parsing system receives the binary header information sent by the file system, it parses the binary header information to obtain the binary index location information and then sends the binary index location information to the file system. The binary index information represents the location information of the index data in binary form.

[0067] The file system reads binary index data from the file to be parsed based on the binary index position information and sends the binary index data to the parsing system. The binary index data represents index data in binary form.

[0068] The parsing system deserializes the binary index data to obtain the index data. The index data stores basic information for multiple data blocks, each including time information, binary position information, and channel information. The binary position information represents the location of the data block in binary form.

[0069] Users can set preset requirements according to their needs, which can include data within a preset time range. The parsing system can filter all data blocks based on the time information of each data block, select target data blocks that meet the preset requirements, and send the binary location information of the target data blocks that meet the preset requirements to the file system.

[0070] After receiving the binary location information of the target data block, the file system reads the data from the target data block that meets the user's requirements based on the binary location information. The data in the target data block is then in binary format, and the target data block is sent to the parsing system.

[0071] Because the basic information of a data block includes data schema information, the original definition of the data within each data block can be determined using this schema information. Based on this original definition, the parsing system can deserialize and parse the data in the target data block to obtain formatted target data. This formatted target data represents a formatted message and can be seamlessly integrated with existing visualization systems for display.

[0072] It is understood that the technical solution provided in this embodiment, through a file parsing system running on the browser, parses the binary header information of the file to be parsed based on the byte position of the pre-obtained binary header information in the file to be parsed, obtaining the binary index position information of the file to be parsed. Based on the binary index position information, the binary index data of the file to be parsed can be read and parsed to obtain the index data of the file to be parsed. Since the index data stores the basic information of multiple data blocks, the target data block that meets the preset requirements can be determined based on the basic information of each data block, and read to obtain the data in the target data block. Then, the data in the target data block is parsed to obtain the formatted target data, which is the data that meets the user's needs. The file to be parsed is a Cyber ​​Record file, so only the target data block that meets the user's needs can be parsed, without parsing the entire Cyber ​​Record file to obtain the formatted target data. Moreover, the parsing can be directly implemented on the browser side, without relying on the pre-stored Cyber ​​Record file in the cloud, and without needing to pre-parse the entire Cyber ​​Record file in the cloud. The record file is then parsed and stored in the cloud, saving the computing resources for parsing data in the cloud and the storage resources for storing the parsed file. This effectively solves the problem of increased cloud data storage and parsing costs caused by pre-parsing and storing a large amount of Cyber ​​Record file data in the cloud in existing technologies.

[0073] In one possible implementation, step S12 involves reading and parsing the binary index data of the file to be parsed based on the binary index position information, including: Based on the binary index position information, the binary index data of the file to be parsed is obtained by reading it from the file system; the binary index data is then deserialized and parsed by the parsing system to obtain the index data.

[0074] Specifically, the file system receives binary index position information sent by the parsing system, reads the file to be parsed based on the binary index position information, obtains the binary index data of the file to be parsed, and returns the binary index data to the parsing system. The binary index data is serialized binary data and needs to be deserialized before visualization can be achieved.

[0075] The parsing system uses protoBuf (protocol buffer) to deserialize the serialized binary index data, that is, to restore the binary format index data back to the original data, thus obtaining the index data. The obtained index data is formatted index information, i.e., the restored index information.

[0076] It can be understood that, since the binary index position information has been parsed, the binary index data of the file to be parsed can be read through the file system based on the binary index position information. Then, through the parsing system, the binary index data can be deserialized and parsed to obtain the index data. This achieves the purpose of obtaining the index data through the parsing system and the file system, so that the target data block that meets the user's requirements can be determined based on the index data.

[0077] In one possible implementation, the basic information includes time information and binary position information. In step S13, based on the basic information of each data block, target data blocks that meet preset requirements are determined and read, including: Step S131: Based on the time information of each data block, determine at least one target data block that meets the preset requirements from multiple data blocks and the reading order of each target data block.

[0078] Step S132: Read the data in the target data block through the file system according to the binary location information of each target data block.

[0079] Specifically, when reading and determining the target data block that meets the preset requirements, the parsed index data includes basic information of multiple data blocks, and the basic information of each data block includes time information and binary position information.

[0080] The time information indicates the time when the data in the data block was recorded, and can be determined based on the timestamps of the data in the data block. The binary position information indicates the position of the data block in the file to be parsed, and is the position information in binary data.

[0081] When users apply data, they may not need to read all data blocks in the file to be parsed; they may only need data within a certain time range. Users can set preset requirements according to their needs, which may include a preset time range.

[0082] In an optional example, the parsing system can determine target data blocks whose timestamps fall within a preset time range, based on the time information of each data block in the index data. Since the time information of the target data blocks can be determined based on the data's time stamp, to ensure that the read or parsed data is ordered chronologically, after determining the target data blocks, all target data blocks can be sorted according to their time information to obtain ordered target data blocks, thereby determining the reading order of each target data block.

[0083] Then, based on the index data, the binary location information of the sorted target data blocks is determined, and the binary location information of the sorted target data blocks is sent to the file system in sequence.

[0084] The file system can read the data from the file to be parsed sequentially from the target data block according to the binary location information of the target data block, following the reading order. At this point, the target data block is data in binary format that meets preset requirements. Because the data is stored in data blocks, the file system can dynamically load the target data block during reading, thereby reducing memory usage. For example, dynamic block loading can reduce memory usage to less than 50MB.

[0085] The file system can also send the target data blocks to the parsing system, so that the parsing system can deserialize the data in the target data blocks read in sequence to obtain the formatted target data.

[0086] In another alternative example, the file system can read each data block from the file to be parsed using the binary location information of each data block and send it to the parsing system. The parsing system then determines the target data block in binary form that meets preset requirements based on the time information of each data block. The parsing system then deserializes the data in the target data block to obtain the formatted target data.

[0087] Understandably, the basic information of a data block includes its time information and binary location information. Based on the time information of each data block, a target data block that meets the preset requirements can be determined from multiple data blocks, along with the reading order of each target data block. For each target data block, according to the reading order, the storage location of the target data block can be found based on its binary location information. The data in the target data block can then be read through the file system. For all target data blocks, the data in all target data blocks can be read according to the reading order. Since the reading order of the target data blocks is determined based on the time information, reading the target data blocks in the reading order ensures that the read data is roughly sorted according to the time information, saving time in subsequent sorting of the structured target data.

[0088] In one possible implementation, the time information includes start and end times; wherein the start and end times are determined based on the timestamps of each data item in the data block; the preset requirements include a preset time range; in step S132, determining at least one target data block that meets the preset requirements and the reading order of each target data block from multiple data blocks based on the time information of each data block includes: For each data block, based on the start time and end time, and a preset time range, it is determined whether the time information of the data block overlaps with the preset time range; when the time information of the data block overlaps with the preset time range, the data block is determined as a target data block; based on the start time of each target data block, the reading order of all the target data blocks is determined.

[0089] Specifically, the index data includes time information for each data block, including the start and end times of the data block. Each data item corresponds to a timestamp. The start time of a data block is determined based on the timestamps of the data within that block; that is, the earliest time determined by the timestamps is the start time of the data block. Similarly, the end time of a data block is determined based on the timestamps of the data within that block; that is, the latest time determined by the timestamps is the end time of the data block.

[0090] The preset requirements include a preset time range, which can be set according to the user's needs. This embodiment does not impose specific limitations on this.

[0091] For each data block in the index data, based on the start and end times of the data block and the preset time range, it can be determined whether the time information of the data block overlaps with the preset time range.

[0092] When the time information of a data block overlaps with a preset time range, the data block is determined to be a target data block. When the time information of a data block does not overlap with the preset time range, the data block is determined not to be a target data block. This allows for the identification of all target data blocks.

[0093] Based on the start time of each target data block, all target data blocks are sorted in ascending order of start time. Based on this sorted list, the reading order of each target data block can be determined.

[0094] Understandably, the time information includes start and end times, which are determined based on the timestamps of each data item in the data block. The preset requirements include a preset time range. For each data block, based on the start and end times and the preset time range, it is determined whether the data block's time information overlaps with the preset time range. When the data block's time information overlaps with the preset time range, the data block is identified as a target data block. This allows target data blocks that meet the preset requirements to be identified from the index data, enabling them to be read from the file system subsequently without needing to read data blocks that do not meet the preset requirements. Furthermore, the start time of each target data block can determine the reading order of all target data blocks, ensuring that the read target data blocks are sorted according to their start times. This helps to ensure that the data parsed later is roughly sorted according to the time information.

[0095] In one possible implementation, the data carries a timestamp; in step S14, the data in the read target data block is parsed to obtain formatted target data, including: Step S141: Filter the data in the target data block according to the timestamp and preset time range to obtain target data that meets the preset requirements; Step S142: The target data is parsed by the parsing system to obtain formatted target data.

[0096] Specifically, each data point carries a timestamp to indicate the data's time of collection.

[0097] After reading all the data from the target data blocks, the data is sorted according to the reading order of the target data blocks. Not all the data may be sorted in chronological order, and there may be data in the target data blocks whose time information does not meet the preset time range.

[0098] For any data point within a target data block, based on its timestamp and a preset time range, determine if the data falls within that preset time range. If the data is within the preset time range, it is determined to be target data. If the data is outside the preset time range, it is determined not to be target data. This process filters the data within the target data block.

[0099] Similarly, the same filtering is applied to all target data blocks to obtain target data that meets the preset requirements.

[0100] The parsing system parses all the target data to obtain formatted target data.

[0101] The data carries a timestamp, and the preset requirements include a preset time range. Based on the timestamp and preset time range of the data in the target data block, the data in the target data block is filtered to obtain target data that meets the preset requirements. The target data is then parsed by the parsing system to obtain formatted target data. By using the method of filtering first and then parsing, the target data can be determined first, and then only the target data needs to be parsed. It is not necessary to parse all the data in all target data blocks, thereby saving the computing power and time costs of parsing.

[0102] In one possible implementation, the basic information includes pattern information corresponding to each data point. In step S142, the target data is parsed by a parsing system to obtain formatted target data, including: Based on the pattern information corresponding to each target data, determine the original definition of each target data; based on the original definition of each target data, determine the target parsing method for each target data; based on the target parsing method for each target data, use the parsing system to deserialize and parse each target data to obtain formatted target data.

[0103] Specifically, the basic information of a data block includes data schema information, such as message definition and SchemaName (data schema name). Based on the data schema information, the binary form of the data can be reconstructed.

[0104] After receiving the target data sent by the file system, the parsing system needs to restore the serialized data since the target data is serialized.

[0105] For any given target data, the parsing system can determine its original definition based on the target data's pattern information. Based on this original definition, the target parsing method can be determined. Using this parsing method, the serialized target data can be deserialized using protoBuf (protocol buffer) to obtain formatted target data. Formatted target data is the data before serialization.

[0106] After deserializing and parsing all the target data using the same method, all the formatted target data is obtained.

[0107] It should be noted that serialized data is deserialized and parsed using protoBuf (protocol buffer), and loaded at runtime using protocol descriptors, which can support the restoration of data in various formats.

[0108] It can be understood that the basic information of each data block includes the pattern information corresponding to the data in that data block. For any target data in each target data block that is read, the original definition of the target data can be determined according to the pattern information corresponding to the target data. According to the original definition, the target parsing method of the target data can be determined. Through the parsing system, the target data can be deserialized and parsed according to the target parsing method. Dynamic decoding can be achieved for all target data without having to decode the entire file to be parsed in advance, so as to obtain the formatted target data that meets the user's needs.

[0109] In one possible implementation, in step S14, the target data block is parsed to obtain formatted target data. Afterwards, the file parsing method further includes: The parsing system sends the formatted target data to the user's terminal device for user use.

[0110] Specifically, after parsing the target data into a formatted format that meets preset requirements, the parsing system returns the formatted data to the user terminal device. The user terminal device can also be a user interface for users to view and use.

[0111] Understandably, the target data block is parsed to obtain formatted target data. Then, the parsing system can send the formatted target data to the user's terminal device for user use.

[0112] In one possible implementation, in step S11, the binary header information of the file to be parsed is parsed. Before this, the above file parsing method also includes: When the file to be parsed is a local file, the binary header information of the file to be parsed is read directly through the file system; when the file to be parsed is a remote file, the binary header information of the file to be parsed is read through the Hypertext Transfer Protocol.

[0113] Specifically, before the parsing system can parse the binary header information of the file to be parsed, it is necessary to read the binary header information of the file to be parsed through the file system, that is, the file system accesses the file header of the file to be parsed. When the file to be parsed is a local file, the binary header information of the file to be parsed can be read directly through the file system. When the file to be parsed is a remote file, the binary header information of the file to be parsed can be read through the Hypertext Transfer Protocol. After reading the binary header information, it is sent to the parsing system so that the parsing system can parse the binary header information to obtain the binary index position information and the basic information of the file to be parsed.

[0114] It should be noted that the basic information of the file to be parsed includes the file name, file size, start time, and end time.

[0115] In one possible implementation, when there are multiple formatted target data, each formatted target data carries a timestamp; step S14 involves parsing the data in the read target data block to obtain the formatted target data. Afterwards, the method further includes: Based on the timestamp, sort the formatted target data to obtain the sorted formatted target data.

[0116] Specifically, when there are multiple formatted target data, each formatted target data carries a timestamp. Based on the timestamps, the chronological order of all formatted target data can be determined. According to the chronological order, all formatted target data can be sorted to obtain sorted formatted target data.

[0117] It is understandable that when there are multiple target data to be formatted, each target data carries a timestamp. Since the target data blocks are sorted data blocks, the target data blocks read are also in order. However, there may be two target data blocks where the target data is not in order. The data in the read target data blocks is parsed to obtain the formatted target data. After that, the formatted target data can be sorted according to the timestamps to obtain the sorted formatted target data.

[0118] In one possible implementation, the above file parsing method also includes: The system monitors for corrupted data in the files to be parsed; when corrupted data is detected, an error message is generated and sent to the user's device.

[0119] Specifically, during the deserialization and parsing of the file to be parsed using protoBuf, errors may occur due to data corruption. An error capture function is added to the parsing system to detect errors during the deserialization process. When an error is detected, it is reported, indicating that the data in the file to be parsed is corrupted at the point of error, thus enabling data corruption monitoring. When corrupted data is detected, an error message can be generated and sent to the user's device for timely review and remedial action.

[0120] Understandably, during the parsing of the file to be parsed, corrupt data is monitored. When corrupt data is found, an error message is generated and sent to the user's device, so that the user can promptly detect and record the corrupt data.

[0121] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0122] Corresponding to the file parsing method described in the above embodiments, Figure 5 The diagram shows a structural schematic of a file parsing system provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0123] Reference Figure 5 The file parsing system 5, running on the browser side, includes a file system 51 and a parsing system 52, wherein: The parsing system 52 is used to parse the binary header information of the file to be parsed, and obtain the binary index position information of the file to be parsed. File system 51 is used to read the binary index data of the file to be parsed based on the binary index position information; The parsing system 52 is also used to parse the binary index data of the file to be parsed to obtain index data; the index data stores basic information of multiple data blocks. The parsing system 52 is also used to determine the target data block that meets the preset requirements based on the basic information of each data block; The file system 51 is also used to read target data blocks that meet preset requirements; The parsing system 52 is also used to parse the data in the read target data block to obtain formatted target data.

[0124] In one possible implementation, the file system 51 is also used to read the binary index data of the file to be parsed based on the binary index position information; The parsing system 52 is also used to deserialize and parse binary index data to obtain index data.

[0125] In one possible implementation, the basic information includes time information and binary position information; The parsing system 52 is also used to determine at least one target data block that meets the preset requirements from multiple data blocks based on the time information of each data block, as well as the reading order of each target data block; the file system 51 is also used to read the data in each target data block according to the binary position information of each target data block and the reading order of each target data block through the file system.

[0126] In one possible implementation, the time information includes a start time and an end time; wherein the start time and end time are determined based on the timestamp of each data in the data block; the preset requirements include a preset time range; the parsing system 52 is further configured to, for each data block, determine whether the time information of the data block overlaps with the preset time range based on the start time and end time and the preset time range; when the time information of the data block overlaps with the preset time range, the data block is determined as a target data block; and the reading order of all target data blocks is determined based on the start time of each target data block.

[0127] In one possible implementation, the data carries a timestamp; the preset requirements include a preset time range; the parsing system 52 is also used to filter the data in the target data block according to the timestamp and the preset time range of the data in the target data block to obtain target data that meets the preset requirements; the parsing system parses the target data to obtain formatted target data.

[0128] In one possible implementation, the basic information includes pattern information corresponding to each data; the parsing system 52 is also used to determine the original definition of each target data according to the pattern information corresponding to each target data; determine the target parsing method of each target data according to the original definition of each target data; and perform deserialization parsing on each target data through the parsing system according to the target parsing method of each target data to obtain formatted target data.

[0129] In one possible implementation, the parsing system 52 is also used to send the formatted target data to the user terminal device for user use.

[0130] In one possible implementation, the binary header information of the file to be parsed is parsed. Previously, the file system 51 was also used to directly read the binary header information of the file to be parsed when the file to be parsed was a local file; and to read the binary header information of the file to be parsed through the Hypertext Transfer Protocol when the file to be parsed was a remote file.

[0131] In one possible implementation, when there are multiple formatted target data, each formatted target data carries a timestamp; the parsing system 52 is also used to sort each formatted target data according to the timestamp to obtain sorted formatted target data.

[0132] In one possible implementation, the parsing system 52 is also used to monitor for corrupted data in the file to be parsed; when corrupted data is found, an error alert message is generated and sent to the user terminal device.

[0133] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0134] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0135] This application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0136] This application also provides a vehicle that includes the file parsing system described above.

[0137] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0138] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0140] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0141] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0142] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A file parsing method, characterized in that, The file parsing method is applied to a file parsing system that runs on a browser and includes: Based on the byte position of the pre-obtained binary header information in the file to be parsed, the binary header information of the file to be parsed is parsed to obtain the binary index position information of the file to be parsed. Based on the binary index position information, the binary index data of the file to be parsed is read and parsed to obtain index data; wherein, the index data stores basic information of multiple data blocks; Based on the basic information of each data block, target data blocks that meet preset requirements are identified and read. The data in the read target data block is parsed to obtain formatted target data.

2. The file parsing method as described in claim 1, characterized in that, The step of reading and parsing the binary index data of the file to be parsed based on the binary index position information includes: Based on the binary index position information, the binary index data of the file to be parsed is obtained by reading it from the file system. The binary index data is deserialized and parsed using a parsing system to obtain the index data.

3. The file parsing method as described in claim 1, characterized in that, The basic information includes time information and binary location information; The step of determining and reading target data blocks that meet preset requirements based on the basic information of each data block includes: Based on the time information of each data block, at least one target data block that meets the preset requirements is determined from the plurality of data blocks, as well as the reading order of each target data block; Based on the binary location information of each target data block, and in the order in which each target data block is read, the data in each target data block is read through the file system.

4. The file parsing method as described in claim 3, characterized in that, The time information includes a start time and an end time; wherein the start time and the end time are determined based on the timestamp of each data item in the data block; The preset requirements include a preset time range; The step of determining at least one target data block that meets preset requirements from a plurality of data blocks based on the time information of each data block, and the reading order of each target data block, includes: For each data block, based on the start time, the end time, and the preset time range, it is determined whether the time information of the data block overlaps with the preset time range; When the time information of the data block overlaps with the preset time range, the data block is determined to be the target data block; The reading order of all the target data blocks is determined based on the start time of each target data block.

5. The file parsing method as described in claim 1, characterized in that, The data carries a timestamp; The preset requirements include a preset time range; The step of parsing the data in the read target data block to obtain formatted target data includes: Based on the timestamps of the data in the target data block and the preset time range, the data in the target data block is filtered to obtain target data that meets the preset requirements; The target data is parsed by the parsing system to obtain the formatted target data.

6. The file parsing method as described in claim 5, characterized in that, The basic information includes the pattern information corresponding to each data point; The target data is parsed by the parsing system to obtain the formatted target data, including: Based on the pattern information corresponding to each target data, determine the original definition of each target data; Based on the original definition of each target data, determine the target parsing method for each target data; According to the target parsing method for each target data, the parsing system performs deserialization parsing on each target data to obtain formatted target data.

7. The file parsing method as described in claim 1, characterized in that, Before parsing the binary header information of the file to be parsed, the method further includes: When the file to be parsed is a local file, the binary header information of the file to be parsed is read directly through the file system; When the file to be parsed is a remote file, the binary header information of the file to be parsed is read through the Hypertext Transfer Protocol.

8. The file parsing method as described in claim 1, characterized in that, When there are multiple formatted target data, each formatted target data carries a timestamp; After parsing the data in the read target data block to obtain formatted target data, the method further includes: Based on the timestamp, each of the formatted target data is sorted to obtain the sorted formatted target data.

9. A file parsing system, running on a browser, characterized in that, The file parsing system includes a file system and a parsing system; wherein: The parsing system is used to parse the binary header information of the file to be parsed based on the byte position of the pre-acquired binary header information in the file to be parsed, so as to obtain the binary index position information of the file to be parsed; wherein, the file to be parsed is a Cyber ​​Record file; The file system is used to read the binary index data of the file to be parsed based on the binary index position information; The parsing system is also used to parse the binary index data of the file to be parsed to obtain index data; wherein, the index data stores basic information of multiple data blocks; The parsing system is also used to determine target data blocks that meet preset requirements based on the basic information of each data block; The file system is also used to read the target data block that meets the preset requirements; The parsing system is also used to parse the data in the read target data block to obtain formatted target data.

10. A vehicle comprising the file parsing system as described in claim 9.

Images