Data analysis method and device

Abstract

The invention discloses a data analysis method which comprises the steps that a data sequence corresponding to source data is obtained, the data sequence is composed of a plurality of data fragments, each data fragment comprises a first data identifier, and the first data identifier is used for representing the type of the data fragment; according to the type of the data fragment, analyzing other data identifiers except the first data identifier in the data fragment to obtain source data; wherein the multiple data fragments comprise a first data fragment, the type of the first data fragment is extended metadata, the first data fragment is located at the tail end of the data sequence, and the first data fragment is used for storing dynamic attribute information of the data sequence.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data parsing method and apparatus. Background Technology

[0002] Drones, unmanned vehicles, and other intelligent devices are being increasingly widely used in logistics and other fields. To improve the stability of these devices in large-scale commercial operations, it is necessary to acquire data from drones and unmanned vehicles during mission execution. Data analysis can then provide a foundation for subsequent equipment upgrades, parameter adjustments, and improvements to operational strategies. However, traditional data analysis platforms primarily focus on file-level data association, lacking the necessary tags and scenario associations to meet the data analysis needs of drone operations, thus failing to link data to specific flight counts or scenarios. Summary of the Invention

[0003] This invention provides a data parsing method and apparatus. By recording operational parameters and task parameters in a specific data format, it enables effective indexing, classification, summarization, and analysis of data files in subsequent data analysis. It achieves accurate restoration of data in the recorded time sequence, supports data encryption, forward and backward data tracking, and random access.

[0004] A first aspect of this invention provides a data parsing method, comprising: obtaining a data sequence corresponding to source data, the data sequence being composed of multiple data segments, each data segment containing a first data identifier, the first data identifier being used to characterize the type of the data segment; parsing other data identifiers in the data segment besides the first data identifier according to the type of the data segment to obtain the source data; wherein, the multiple data segments include a first data segment, the type of the first data segment being extended metadata, the first data segment being located at the end of the data sequence, and the first data segment being used to store dynamic attribute information of the data sequence.

[0005] In one feasible implementation, other data identifiers for the first data segment include a first length identifier and a first content identifier, wherein the first length identifier is used to characterize the data length of the first content identifier, and the first content identifier is used to store the dynamic attribute information.

[0006] In one feasible implementation, the dynamic attribute information includes descriptive information that changes during the recording task of the source data.

[0007] In one feasible implementation, the dynamic attribute information includes one or more of the following: the recording duration of the source data, the statistical information of the source data, and the process information contained in the source data.

[0008] In one feasible implementation, the plurality of data segments include a second data segment, the type of which is basic metadata, and the second data segment is used to store static attribute information of the data sequence; other data identifiers of the second data segment include a second length identifier and a second content identifier, the second length identifier being used to characterize the data length of the second content identifier, and the second content identifier being used to store the static attribute information.

[0009] In one feasible implementation, the static attribute information includes descriptive information that remains unchanged during the recording task of the source data.

[0010] In one feasible implementation, the static attribute information includes one or more of the following: the identity information of the data sequence, the device information for recording the source data, the start recording time of the source data, and the encryption information of the data sequence.

[0011] In one feasible implementation, the plurality of data segments includes a third data segment, the type of which is an index table, and the third data segment is used to store a list of content topics corresponding to the source data; other data identifiers of the third data segment include a third length identifier and a third content identifier, the third length identifier being used to characterize the data length of the third content identifier, and the third content identifier being used to store the list.

[0012] In one feasible implementation, the plurality of data segments includes a fourth data segment, the type of which is a message. The fourth data segment is used to store the content segment of the source data corresponding to the fourth data segment. Other data identifiers of the fourth data segment include a header information length identifier, a header information identifier, a message length identifier, a message content identifier, and a prior data position identifier. The header information length identifier is used to characterize the data length of the header information and to store the tag information of the content segment. The message length identifier is used to characterize the data length of the message content identifier and to store the content segment. The prior data position identifier is used to store the position information of the previous data segment in the data sequence in the recording time order of the source data.

[0013] In one feasible implementation, the plurality of data segments includes a fifth data segment, the type of which is a delayed message, and the fifth data segment is used to store the content segment of the source data corresponding to the fifth data segment; other data identifiers of the fifth data segment include the header information length identifier, the header information identifier, the message length identifier, and the message content identifier.

[0014] In one feasible implementation, the tag information includes topic information of the content fragment, which is used to determine the content topic of the content fragment of the source data in the list.

[0015] In one feasible implementation, the plurality of data segments includes a sixth data segment, the sixth data segment being of type message pointer, the sixth data segment being used to indicate the position of the fifth data segment in the data sequence; other data identifiers of the sixth data segment include a delay information position identifier and a prior data position identifier, the delay information position identifier being used to store the position information of the fifth data segment in the data sequence.

[0016] In one feasible implementation, after parsing other data identifiers in the data segment besides the first data identifier according to the type of the data segment to obtain the source data, the method further includes: parsing extended metadata location identifiers to obtain the position of the first data segment in the data sequence.

[0017] In one feasible implementation, before parsing other data identifiers in the data segment besides the first data identifier according to the type of the data segment to obtain the source data, the method further includes: parsing the first flag bit to obtain the data format of the data sequence.

[0018] In one feasible implementation, after parsing the extended metadata location identifier to obtain the position of the first data fragment in the data sequence, the method further includes: parsing a second flag bit, the second flag bit being an end identifier of the data sequence.

[0019] A second aspect of the present invention provides a data recording method, comprising: determining static attribute information of source data and incorporating the static attribute information into a data sequence as a second data segment; obtaining a list of content topics corresponding to the source data and incorporating the list into the data sequence as a third data segment; sequentially obtaining content segments of the source data and incorporating the content segments into the data sequence as a fourth data segment; determining dynamic attribute information of the source data and incorporating the dynamic attribute information into the data sequence as a first data segment; determining the starting position of the first data segment in the data sequence and incorporating the starting position into the data sequence as an extended metadata position identifier.

[0020] In one feasible implementation, the content segment includes a first content segment and a second content segment. The step of sequentially acquiring the content segments of the source data and incorporating the content segments into the data sequence as a fourth data segment includes: acquiring the first content segment; performing data processing on the first content segment; incorporating it into the data sequence as a sixth data segment; acquiring the second content segment; incorporating the second content segment into the data sequence as the fourth data segment; incorporating the data-processed first content segment into the data sequence as a fifth data segment, and incorporating the position of the fifth data segment in the data sequence into the sixth data segment using a delay information position identifier.

[0021] In one feasible implementation, the data identifier of the first data segment includes a first data identifier, a first length identifier, and a first content identifier. The first data identifier is used to characterize the type of the first data segment as extended metadata, the first length identifier is used to characterize the data length of the first content identifier, and the first content identifier is used to store the dynamic attribute information.

[0022] In one feasible implementation, the dynamic attribute information includes descriptive information that changes during the recording task of the source data.

[0023] In one feasible implementation, the dynamic attribute information includes one or more of the following: the recording duration of the source data, the statistical information of the source data, and the process information contained in the source data.

[0024] In one feasible implementation, the data identifier of the second data segment includes a first data identifier, a second length identifier, and a second content identifier. The first data identifier is used to characterize the type of the second data segment as basic metadata, the second length identifier is used to characterize the data length of the second content identifier, and the second content identifier is used to store the static attribute information.

[0025] In one feasible implementation, the static attribute information includes descriptive information that remains unchanged during the recording task of the source data.

[0026] In one feasible implementation, the static attribute information includes one or more of the following: the identity information of the data sequence, the device information for recording the source data, the start recording time of the source data, and the encryption information of the data sequence.

[0027] In one feasible implementation, the data identifier of the third data segment includes a first data identifier, a third length identifier, and a third content identifier. The first data identifier is used to characterize the type of the third data segment as an index table, the third length identifier is used to characterize the data length of the third content identifier, and the third content identifier is used to store the list.

[0028] In one feasible implementation, the data identifier of the fourth data segment includes a first data identifier, a header information length identifier, a header information identifier, a message length identifier, a message content identifier, and a prior data position identifier. The first data identifier is used to characterize the type of the fourth data segment as a message. The header information length identifier is used to characterize the data length of the header information. The header information identifier is used to store the tag information of the content segment. The message length identifier is used to characterize the data length of the message content identifier. The message content identifier is used to store the content segment. The prior data position identifier is used to store the position information of the previous data segment in the data sequence according to the recording time order of the source data.

[0029] In one feasible implementation, the data identifier of the fifth data segment includes the first data identifier, the header information length identifier, the header information identifier, the message length identifier, and the message content identifier; the first data identifier is used to characterize the type of the fifth data segment as delay information.

[0030] In one feasible implementation, the tag information includes topic information of the content fragment, which is used to determine the content topic of the content fragment of the source data in the list.

[0031] In one feasible implementation, the data identifier of the sixth data segment includes the first data identifier, the delay information location identifier, and the prior data location identifier. The first data identifier is used to characterize the type of the sixth data segment as a message pointer, and the delay information location identifier is used to store the location information of the fifth data segment in the data sequence.

[0032] In one feasible implementation, before incorporating the static attribute information into the data sequence as a second data segment, the method further includes: incorporating a first flag bit into the data sequence, the first flag bit being used to indicate the data format of the data sequence.

[0033] In one possible implementation, after incorporating the start position into the data sequence as an extended metadata location identifier, the method further includes incorporating a second flag bit into the data sequence, the second flag bit being used to indicate the end of recording in the data sequence.

[0034] A third aspect of the present invention discloses a data processing apparatus, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute method steps according to any one of the first or second aspects under the control of the computer program.

[0035] A fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method steps according to any one of the first or second aspects.

[0036] This invention, through the design of a specific data structure, associates more dimensions of information in the collected data with the collected data in a label-like manner, making the data analysis more comprehensive and better meeting the actual operational needs for data analysis.

[0037] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0039] Figure 1 This is a schematic diagram of the overall architecture of a UAV data loopback provided in an embodiment of this application;

[0040] Figure 2 This is a flowchart illustrating a data parsing method provided in an embodiment of the present invention;

[0041] Figure 3A This is a schematic diagram of a data structure provided in an embodiment of the present invention;

[0042] Figure 3B This is a schematic diagram of another data structure provided in an embodiment of the present invention;

[0043] Figure 4 This is a flowchart illustrating a data recording method provided in an embodiment of the present invention;

[0044] Figure 5 This is an exemplary structural block diagram of a device provided in an embodiment of the present invention. Detailed Implementation

[0045] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0046] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0047] Techniques, methods, and apparatus known to those skilled in the art in the relevant field may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

[0048] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0049] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0050] The embodiments of the present invention can be applied to various data analysis-related scenarios. For ease of description, this article only uses the drone operation scenario as an example for description.

[0051] The UAV data processing system includes data acquisition, data uploading, data management, and data application. This paper refers to the operation process of each of these stages as the "data loop." The data loop constructs a complete data tool system platform for UAVs and their related intelligent hardware devices. Its main purpose is to ensure that data can be transmitted efficiently and accurately from the device to the data center, and after processing and analysis, provide valuable information for algorithm engineers, R&D personnel, and testers.

[0052] Specifically, data acquisition primarily involves collecting raw data from drones or other smart hardware. This collected data, known as loopback data, includes location information, flight status, video image data, and other raw sensor data. This data directly reflects the operational status and performance of the drone or smart hardware. The collected loopback data needs to be transmitted to the drone's data center in a timely and secure manner. This process may require consideration of various factors, such as data transmission speed, data security to prevent interception or tampering, and transmission stability. The drone's data center manages the loopback data. In the drone's data center, the received loopback data needs to be stored, categorized, and organized to ensure efficient access and utilization, while also facilitating subsequent data analysis and processing. Data application is the ultimate goal of data loopback. Through data analysis and processing, algorithm engineers and developers can understand the operational status of the drone or smart hardware, identify potential problems, evaluate the quality of algorithms, and optimize algorithm models accordingly. This not only improves the performance of the drone or smart hardware but also ensures their operational safety and reliability.

[0053] like Figure 1 As shown, the overall architecture of the UAV data loop is divided into three parts: the terminal, the edge, and the cloud. The terminal realizes data collection and filtering; the edge realizes data download and upload; and the cloud realizes data management, indexing, and data application, including truth systems, simulation and regression test quality assessment, etc.

[0054] The drone loopback data originates from the business scenarios of drone operations. Drones, airport equipment, and various sensor hardware are the sources of data generation. In testing and delivery scenarios, the loopback data can be correlated and indexed based on specific routes, tasks, and environmental conditions.

[0055] The ultimate goal of storing and managing UAV loopback data is data application. Basic application scenarios require the ability to accumulate loopback datasets over a long period, providing structured retrieval and application scenario association capabilities to form data assets, supporting on-demand batch data extraction, accumulating various test and verification sample sets for complex scenarios, improving the efficiency of problem localization and resolution, and providing scenario data support for the training and evaluation of algorithm models.

[0056] The storage and management of drone loopback data relies on cloud-based data centers. These data centers offer functions such as multi-dimensional indexing and retrieval of log files, real-time display of operational sorties, visualization of aggregated metrics from multiple algorithms and sensor modules, and compliance data management, providing platform support for log file collection and management. However, data association methods are primarily focused on the file dimension, lacking tag and scenario association, and thus unable to link data to sorties or specific scenarios.

[0057] To address the aforementioned issues, embodiments of the present invention relate to a data format that facilitates scene correlation analysis and a method for recording and parsing it. Figure 2 An exemplary data parsing method S200 is provided, which includes:

[0058] S201. Obtain the data sequence corresponding to the source data.

[0059] The data sequence consists of multiple data segments, each of which contains a first data identifier that is used to characterize the type of the data segment.

[0060] The source data, also known as the loopback data mentioned above, can, for example, include raw data obtained during the data acquisition phase from various acquisition devices such as sensors, measuring instruments, cameras, and data loggers. The data sequence is a data file that records this raw data in a specific data format.

[0061] It should be understood that a data sequence can be divided into multiple data segments. Some of these segments contain auxiliary information for parsing other data segments, some contain attribute information of the data sequence and the corresponding source data, and some contain content segments of the source data. To distinguish these data segments with different functions, each data segment needs to contain a first data identifier to characterize the type of that data segment.

[0062] S202. Based on the type of the data segment, parse the other data identifiers in the data segment besides the first data identifier to obtain the source data.

[0063] In this embodiment of the invention, different types of data fragments may have different syntactic structures. After the first data identifier representing the type of the data fragment is parsed, the location and parsing method of the other data identifiers of the data fragment are determined, and the content of the other data identifiers can be determined according to a preset method. Once all data fragments have been parsed, the source data can be obtained.

[0064] In one feasible implementation, the plurality of data fragments includes a first data fragment of type extended metadata. This first data fragment is located at the end of the data sequence and is used to store dynamic attribute information of the data sequence. That is, the data fragment type includes extended metadata; in this document, data fragments of type extended metadata are collectively referred to as the first data fragment.

[0065] In one feasible implementation, the dynamic attribute information includes descriptive information that changes during the recording task of the source data; the dynamic attribute information includes one or more of the following: the recording duration of the source data, the statistical information of the source data, and the process information contained in the source data. It should be understood that dynamic attribute information exists during the recording process of the source data, but can only be determined when the source data is completely recorded; it is contextual information of the source data and may change during the recording process, therefore the size of the data is also uncertain. Dynamic attribute information is not limited to combinations of one or more of the above-mentioned information, but may also include other information with the same properties.

[0066] For example, dynamic attribute information can include business attribute information, tag attribute information, and status attribute information. Business attribute information includes environmental and business-related information generated by the loopback data, such as associated flight number, task type, operating environment, operating scenario, and associated flight route. Tag attribute information includes automatic tags generated by the device during the loopback data collection process based on business understanding, such as error codes and scenario tags. Status attribute information includes transmission progress and decompression progress, which are used for human-machine interaction logic control in data management and can support functions such as basic data retrieval, decompression and extraction of frame data, comprehensive analysis, and data playback.

[0067] In drone operation scenarios, drone operation information can also be used as dynamic attribute information, such as flight information during drone delivery missions, including geographical location, commercial area identification, flight trajectory identification, and alarm codes that occur during the recording of delivery mission data. It can also include statistical data for all recorded topics, such as the timestamps of the first and last messages in each topic, the number of messages in each topic, and the recording frame rate.

[0068] In one feasible implementation, the other data identifiers of the first data segment include a first length identifier and a first content identifier. The first length identifier is used to characterize the data length of the first content identifier, and the first content identifier is used to store the dynamic attribute information. An embodiment of the present invention provides an exemplary data structure for the first data segment:

[0069] |First data identifier (1 byte)|First length identifier ({X} bytes)|First content identifier ({first length identifier} bytes)|

[0070] The first data identifier (1 byte) means that the first data identifier is recorded in 1 byte, and the first length identifier ({X} bytes) means that the first length identifier is recorded in X bytes. Other similar descriptions in the following text can be referred to this example for explanation, and will not be repeated here.

[0071] It should be understood that since the content of dynamic attribute information cannot be predetermined, X cannot be determined in advance either. In some embodiments, the content of the first length identifier is determined using a preset data format. For example, sufficient fixed-length encoding space can be reserved for the first length identifier, or a variable-length encoding method can be used to determine the value of X.

[0072] In one feasible implementation, the plurality of data fragments includes a second data fragment of type basic metadata, which is used to store static attribute information of the data sequence. That is, the data fragment type includes basic metadata; in this document, data fragments of type basic metadata are collectively referred to as second data fragments.

[0073] In one feasible implementation, the static attribute information includes descriptive information that remains unchanged throughout the recording task of the source data. The static attribute information includes one or more of the following: the identity information of the data sequence, the device information for recording the source data, the start recording time of the source data, and the encryption information of the data sequence. The static attribute information may also include calibration data necessary for interpreting and processing sensor samples. It should be understood that static attribute information is contextual information that is available when the source data begins recording and remains unchanged throughout the recording process. Compared to dynamic attribute information, the data length of static attribute information can be predetermined. Static attribute information is not limited to combinations of one or more of the above-mentioned information and may also include other information with the same properties. Setting static attribute information as a separate type of data segment avoids the need for updating this type of data after it has been written into the data sequence.

[0074] For example, static attribute information can include file self-attribute information and device attribute information. Self-attribute information represents the basic attributes of the original text of the loopback data itself, such as a unique identifier that distinguishes it from other data sequences in the data loop, creation time, file size, format version, encryption algorithm version, integrity check code, etc. Device attribute information includes device information generated by the loopback data, such as device serial number, device type, device name, device batch number, software and hardware version number, etc.

[0075] Static and dynamic attribute information together provide an overview of the source data for analysis, and also provide rich classification information for creating various types of indexes. As the indexes are effectively established, access to all dimensions of the data sequence becomes more convenient.

[0076] In one feasible implementation, the other data identifiers of the second data segment include a second length identifier and a second content identifier. The second length identifier is used to characterize the data length of the second content identifier, and the second content identifier is used to store the static attribute information. An embodiment of the present invention provides an exemplary data structure for the second data segment:

[0077] |First data identifier (1 byte)|Second length identifier (4 bytes)|Second content identifier ({second length identifier} bytes)|

[0078] In one feasible implementation, the plurality of data fragments includes a third data fragment of type index table, which is used to store a list of content topics corresponding to the source data. That is, the data fragment type includes index tables; in this document, data fragments of type index table are collectively referred to as third data fragments.

[0079] Data segments of type index table are used to provide a complete list of topics involved in the source data. In some embodiments, the list contains a list of all topics pre-defined in the data loop; in other embodiments, the list only contains a list of topics involved in the data sequence in which the third data segment is located; and in still other embodiments, the list of topics included in the list is controlled by data identifiers or training methods located outside the third data segment.

[0080] In one feasible implementation, the other data identifiers of the third data segment include a third length identifier and a third content identifier. The third length identifier is used to characterize the data length of the third content identifier, and the third content identifier is used to store the list. An embodiment of the present invention provides an exemplary data structure for the third data segment:

[0081] |First data identifier (1 byte)|Third length identifier (4 bytes)|Third content identifier ({third length identifier} bytes)|

[0082] The third internal identifier stores strings for each topic in the list, or preset codes or abbreviations. By introducing independent index table data fragments, the topic name is uniformly stored for each subsequent data fragment recording topic information, improving storage efficiency and data readability for data access personnel.

[0083] In one feasible implementation, the plurality of data segments includes a fourth data segment, which is of type message. This fourth data segment is used to store the content segment of the source data corresponding to it. That is, the data segment type includes messages. In this document, data segments of type message are collectively referred to as the fourth data segment.

[0084] In one feasible implementation, the other data identifiers of the fourth data segment include a header length identifier, a header identifier, a message length identifier, a message content identifier, and a prior data position identifier. The header length identifier characterizes the data length of the header information and stores the tag information of the content segment. The message length identifier characterizes the data length of the message content identifier and stores the content segment. The prior data position identifier stores the position information of the previous data segment in the data sequence according to the recording time order of the source data. An embodiment of the present invention provides an exemplary data structure for the fourth data segment:

[0085] | First Data Identifier (1 byte) | Header Length Identifier (2 bytes) | Header Identifier ({Header Length Identifier} bytes) | Message Length Identifier (4 bytes) | Message Content Identifier ({Message Length Identifier} bytes) | First Data Position Identifier (8 bytes)

[0086] The tag information includes the topic information of the content fragment, which is used to identify the topic of the content fragment of the source data in the list. That is, each message type data fragment will include the tag information of the corresponding source data fragment content in its header information; the tag information can be an index of the topic information. Data accessors can then parse this index to determine the topic information of the source data fragment content from a third data fragment of type index table. In some embodiments, the tag information may also include other auxiliary information, such as the recording time information of the source data fragment content. For example, the pseudocode form of the header information identification part can be represented as:

[0087] Information type header data {

[0088] 32-bit unsigned integer index for topic name = 1; / / Index value of the topic list in the topic list data fragment.

[0089] 64-bit unsigned integer record time = 2; / / System time when recording information, in nanoseconds};

[0090] By binding the content of source data fragments within a data segment to its theme, the technical problem of data lacking labels and scene associations, and being unable to be associated with specific flight or scene dimensions, can be solved. Simultaneously, storing the recording time of the source data fragment content in the header information also facilitates access and processing of time-series frames. The prior data position identifier stores the position of the previous data fragment in the data sequence. During data acquisition, the content of data fragments is recorded chronologically; therefore, when scanning the data sequence file from beginning to end, all recorded messages can be traversed in the order they were recorded. The existence of the prior data position identifier makes backward traversal possible, facilitating the examination of data within a specific time window of the recording time and providing convenience for tracing back and forth.

[0091] In one possible application scenario, the temporary data storage space (such as flash memory) of data acquisition devices is limited, but various data acquisition devices such as sensors and measuring instruments are abundant. Recording large amounts of acquired data in an efficient format can not only obtain more valuable data but also help save data transmission costs. For example, after acquiring video content, it needs to be compressed and encoded to save storage space. Therefore, before storing the video content in the form of a data sequence according to the data format in this embodiment of the invention, the video content usually needs to be encoded and compressed asynchronously on a hardware encoder. Before the encoding and compression are completed, other source data may need to be recorded. In this case, the arrival order and / or recording time information of the source data that needs additional processing can be temporarily recorded first, and other source data that can be immediately stored in the data sequence can be recorded instead of waiting for the completion of the previous source data processing flow.

[0092] To address the aforementioned issues, in some feasible implementations, the plurality of data segments includes a fifth data segment, which is of type delayed message. This fifth data segment is used to store the content segment of the source data corresponding to it. That is, the data segment type includes delayed messages; in this document, data segments of type delayed message are collectively referred to as the fifth data segment. The data structure of the fifth data segment is similar to that of the fourth data segment, but it does not contain a prior data position identifier. An embodiment of the present invention provides an exemplary data structure for the fifth data segment:

[0093] | First data identifier (1 byte) | Header length identifier (2 bytes) | Header identifier ({header length identifier} bytes) | Message length identifier (4 bytes) | Message content identifier ({message length identifier} bytes)

[0094] In one feasible implementation, in conjunction with the fifth data segment, the plurality of data segments include a sixth data segment, the sixth data segment being of the type of message pointer, the sixth data segment being used to indicate the position of the fifth data segment in the data sequence; other data identifiers of the sixth data segment include a delay information position identifier and a prior data position identifier, the delay information position identifier being used to store the position information of the fifth data segment in the data sequence.

[0095] The sixth data segment corresponds one-to-one with the fifth data segment and precedes the fifth data segment in the data sequence. During data recording, the source data of the fifth data segment, which is still being processed, can be temporarily skipped, and only the sixth data segment is used to reserve a position for the fifth data segment without waiting for its processing time. The actual recording time order of the source data content segments corresponding to the fifth data segment can be restored by sequentially accessing the sixth data segment and its corresponding fifth data segment in the data sequence. During data parsing, when the sixth data segment is parsed, the fifth data segment indicated by its delay information position identifier can be obtained, and the content segment of the source data corresponding to that fifth data segment can be parsed. When the fifth data segment is parsed, it can be skipped directly because there must have been a parsing process for the corresponding sixth data segment before it; as mentioned earlier, the content segment of the source data corresponding to the fifth data segment has already been parsed. The prior data position identifier in the sixth data segment has the same function as its function in the fourth data segment, and will not be repeated here. An exemplary data structure for the sixth data segment is given in this embodiment of the invention:

[0096] | First data identifier (1 byte) | Delay information position identifier (8 bytes) | Prior data position identifier (8 bytes)

[0097] In one feasible implementation, after parsing the data identifiers other than the first data identifier in the data segment according to the type of the data segment to obtain the source data, the method further includes:

[0098] S203. Parse the extended metadata location identifier to obtain the position of the first data segment in the data sequence. When accessing data in a forward-to-back order, the first data segment of type extended metadata can be determined by parsing according to the data structure order. However, when accessing data in a backward-to-forward order, since the data length of the first data segment is uncertain, the extended metadata location identifier is needed to indicate the starting position of the first data segment to complete the parsing of the first data segment. In other words, setting the extended metadata location identifier improves the convenience of bidirectional data access. For example, the extended metadata location identifier can be an 8-bit low-order byte.

[0099] In one feasible implementation, before parsing other data identifiers in the data segment besides the first data identifier according to the type of the data segment to obtain the source data, the method further includes:

[0100] S204. Parse the first flag bit to obtain the data format of the data sequence.

[0101] It should be understood that data formats are diverse, and even within the same application environment, there may be needs and situations where different data formats coexist. Therefore, a data sequence needs its own unique identifier to indicate its data type. In this embodiment of the invention, the first flag bit is used to indicate that the source data is parsed using the data parsing method of this embodiment. Furthermore, after obtaining the first flag bit through parsing, the identifier of the subsequent data to be parsed can be determined based on the first flag bit.

[0102] In one feasible implementation, the first flag bit is a preset fixed number of 7 bytes. For example, the first flag bit could be 0x640x610x740x610x760x6f 0x78. In another feasible implementation, the first flag bit is a preset fixed number of 8 bytes to facilitate bit alignment. In yet another feasible implementation, the first flag bit is an 8-byte number, where the first 7 bytes are a preset fixed number and the 8th byte represents the version number of the parsing method. It should be understood that different parsing method versions can also be considered as different parsing methods; in this case, the first flag bit is still used to determine the data format and parsing method of the data sequence.

[0103] In one feasible implementation, after parsing the extended metadata location identifier to obtain the position of the first data fragment in the data sequence, the method further includes:

[0104] S205. Parse the second flag bit, which is the end identifier of the data sequence.

[0105] Like the first flag, the second flag can be a preset fixed number of 7 bytes, or another preset number format. If the data sequence is correctly recorded and closed during the recording process, then the second flag should be used as the last part of the data sequence. If the second flag is missing, it means that the data sequence is likely incomplete or corrupted.

[0106] Figure 3AAn exemplary data structure corresponding to the data parsing method of this invention is provided. From this data structure, it can be seen that the data segments are arranged sequentially to form a data sequence. In one feasible implementation, the first flag (including the version number) and the second flag are located at the beginning and end of the data sequence, respectively. Adjacent to the first flag is a data segment of the basic metadata type, followed by a data segment of the index table type. There is no special stipulation regarding the order of the data segments of the basic metadata type and the index table type. Following this are multiple data segments of message types and delayed message types. The delayed message type data segments are pointed to by data segments of message pointer types and are located after the message pointer type data segments. Both message type and message pointer type data segments are capable of tracing back to their preceding data segment in the chronological order of the source data record. The extended metadata type data segment is located at the end of the entire data sequence, followed by an extended metadata position identifier to indicate the position of the extended metadata type data segment. Figure 3B An alternative data structure corresponding to the data parsing method of this embodiment of the invention is provided as an example. From this data structure, it can be seen that the data segments are arranged sequentially to form a data sequence. In one feasible implementation, the first flag bit (including the version number) and the second flag bit are located at the beginning and end of the data sequence, respectively. Adjacent to the first flag bit is a data segment of the basic metadata type, followed by a data segment of the index table type. There is no special stipulation regarding the order of the data segments of the basic metadata type and the data segments of the index table type. Following this are multiple data segments of the message type. The message type data segments are capable of tracing back to their preceding data segment in the chronological order of the source data record. The data segment of the extended metadata type is located at the end of the entire data sequence, followed by an extended metadata position identifier to indicate the position of the extended metadata type data segment.

[0107] Figure 4 An exemplary data recording method S400 is provided. This method records the collected source data into a data sequence according to a specific data format. This data recording method can also be called a data encoding method. It should be understood that, generally, the data encoding process is the reverse of the data parsing process. The encoding steps in method S400, which are completely inverses of the parsing steps, can be referenced in the data parsing method S200 for their implementation and function, and will not be elaborated further here. Method S400 includes:

[0108] S401. Determine the static attribute information of the source data and incorporate the static attribute information into the data sequence as a second data segment;

[0109] S402. Obtain a list of content topics corresponding to the source data, and incorporate the list into the data sequence as a third data segment;

[0110] S403. Sequentially acquire the content segments of the source data, and encode the content segments into the data sequence as the fourth data segment;

[0111] S404. Determine the dynamic attribute information of the source data, and incorporate the dynamic attribute information into the data sequence as a first data segment;

[0112] S405. Determine the starting position of the first data segment in the data sequence, and incorporate the starting position into the data sequence with an extended metadata position identifier.

[0113] In one feasible implementation, the content segment includes a first content segment and a second content segment. The step of sequentially acquiring the content segments of the source data and incorporating the content segments into the data sequence as a fourth data segment includes: acquiring the first content segment; performing data processing on the first content segment; incorporating it into the data sequence as a sixth data segment; acquiring the second content segment; incorporating the second content segment into the data sequence as the fourth data segment; incorporating the data-processed first content segment into the data sequence as a fifth data segment, and incorporating the position of the fifth data segment in the data sequence into the sixth data segment using a delay information position identifier.

[0114] In one feasible implementation, the data identifier of the first data segment includes a first data identifier, a first length identifier, and a first content identifier. The first data identifier is used to characterize the type of the first data segment as extended metadata, the first length identifier is used to characterize the data length of the first content identifier, and the first content identifier is used to store the dynamic attribute information.

[0115] In one feasible implementation, the dynamic attribute information includes descriptive information that changes during the recording task of the source data.

[0116] In one feasible implementation, the dynamic attribute information includes one or more of the following: the recording duration of the source data, the statistical information of the source data, and the process information contained in the source data.

[0117] In one feasible implementation, the data identifier of the second data segment includes the first data identifier, the second length identifier, and the second content identifier. The first data identifier is used to characterize the type of the second data segment as basic metadata, the second length identifier is used to characterize the data length of the second content identifier, and the second content identifier is used to store the static attribute information.

[0118] In one feasible implementation, the static attribute information includes descriptive information that remains unchanged throughout the recording task of the source data.

[0119] In one feasible implementation, the static attribute information includes one or more of the following: the identity information of the data sequence, the device information for recording the source data, the start recording time of the source data, and the encryption information of the data sequence.

[0120] In one feasible implementation, the data identifier of the third data segment includes the first data identifier, the third length identifier, and the third content identifier. The first data identifier is used to indicate that the type of the third data segment is an index table, the third length identifier is used to indicate the data length of the third content identifier, and the third content identifier is used to store the list.

[0121] In one feasible implementation, the data identifier of the fourth data segment includes the first data identifier, header information length identifier, header information identifier, message length identifier, message content identifier, and prior data position identifier. The first data identifier is used to indicate that the type of the fourth data segment is a message. The header information length identifier is used to indicate the data length of the header information. The header information identifier is used to store the tag information of the content segment. The message length identifier is used to indicate the data length of the message content identifier. The message content identifier is used to store the content segment. The prior data position identifier is used to store the position information of the previous data segment in the data sequence in the recording time order of the source data.

[0122] In one feasible implementation, the data identifier of the fifth data segment includes the first data identifier, the header information length identifier, the header information identifier, the message length identifier, and the message content identifier; the first data identifier is used to characterize that the type of the fifth data segment is delayed information.

[0123] In one feasible implementation, the tag information includes topic information of the content fragment, which is used to determine the content topic of the content fragment of the source data in the list.

[0124] In one feasible implementation, the data identifier of the sixth data segment includes the first data identifier, the delay information location identifier, and the prior data location identifier. The first data identifier is used to characterize the type of the sixth data segment as a message pointer, and the delay information location identifier is used to store the location information of the fifth data segment in the data sequence.

[0125] In one feasible implementation, before incorporating the static attribute information into the data sequence as a second data segment, the method further includes:

[0126] S406. The first flag bit is encoded into the data sequence, and the first flag bit is used to indicate the data format of the data sequence.

[0127] In one feasible implementation, after incorporating the starting position into the data sequence using an extended metadata location identifier, the method further includes:

[0128] S407. The second flag bit is encoded into the data sequence, which is used to indicate the end of the record in the data sequence.

[0129] Figure 5 An exemplary structural block diagram of a data processing device 500 is provided, which can be a data parsing device, a data recording device, or a data encoding device. The device 500 includes a memory 501 and a processor 502. The memory 501 stores a computer program, and the processor 502, under the control of the computer program, executes functions such as... Figure 2 or Figure 4 The method steps shown in any one of them.

[0130] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0131] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0132] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device acquires the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0133] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0134] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0135] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0136] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0138] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A data parsing method, characterized in that, include: Obtain the data sequence corresponding to the source data. The data sequence consists of multiple data segments, and each data segment contains a first data identifier, which is used to characterize the type of the data segment. Based on the type of the data segment, parse the data identifiers other than the first data identifier in the data segment to obtain the source data; The plurality of data segments include a first data segment, the type of which is extended metadata. The first data segment is located at the end of the data sequence and is used to store dynamic attribute information of the data sequence.

2. The method according to claim 1, characterized in that, Other data identifiers for the first data segment include a first length identifier and a first content identifier. The first length identifier is used to characterize the data length of the first content identifier, and the first content identifier is used to store the dynamic attribute information. And / or, The dynamic attribute information includes descriptive information that changes during the recording task of the source data; And / or, The dynamic attribute information includes one or more of the following: the recording duration of the source data, the statistical information of the source data, and the process information contained in the source data.

3. The method according to claim 1, characterized in that, The plurality of data segments include a second data segment, the type of which is basic metadata, and the second data segment is used to store static attribute information of the data sequence; other data identifiers of the second data segment include a second length identifier and a second content identifier, the second length identifier being used to characterize the data length of the second content identifier, and the second content identifier being used to store the static attribute information; And / or, The static attribute information includes descriptive information that remains unchanged throughout the recording task of the source data; And / or, The static attribute information includes one or more of the following: the identity information of the data sequence, the device information for recording the source data, the start recording time of the source data, and the encryption information of the data sequence.

4. The method according to claim 1, characterized in that, The plurality of data segments include a third data segment, the type of which is an index table. The third data segment is used to store a list of content topics corresponding to the source data. Other data identifiers of the third data segment include a third length identifier and a third content identifier. The third length identifier is used to characterize the data length of the third content identifier, and the third content identifier is used to store the list.

5. The method according to claim 4, characterized in that, The plurality of data segments includes a fourth data segment, the type of which is a message. The fourth data segment is used to store the content segment of the source data corresponding to the fourth data segment. Other data identifiers of the fourth data segment include a header information length identifier, a header information identifier, a message length identifier, a message content identifier, and a prior data position identifier. The header information length identifier is used to characterize the data length of the header information and to store the tag information of the content segment. The message length identifier is used to characterize the data length of the message content identifier and to store the content segment. The prior data position identifier is used to store the position information of the preceding data segment in the data sequence in the recording time order of the source data.

6. The method according to claim 5, characterized in that, The plurality of data segments includes a fifth data segment, the type of which is a delayed message, and the fifth data segment is used to store the content segment of the source data corresponding to the fifth data segment; other data identifiers of the fifth data segment include the header information length identifier, the header information identifier, the message length identifier, and the message content identifier; And / or, The tag information includes the topic information of the content fragment, which is used to determine the content topic of the content fragment of the source data in the list.

7. The method according to claim 6, characterized in that, The plurality of data segments includes a sixth data segment, the type of which is a message pointer, and the sixth data segment is used to indicate the position of the fifth data segment in the data sequence; other data identifiers of the sixth data segment include a delay information position identifier and a prior data position identifier, the delay information position identifier being used to store the position information of the fifth data segment in the data sequence.

8. The method according to claim 2, characterized in that, After parsing other data identifiers in the data segment besides the first data identifier according to the type of the data segment to obtain the source data, the method further includes: parsing extended metadata location identifiers to obtain the position of the first data segment in the data sequence.

9. The method according to claim 8, characterized in that, Before parsing other data identifiers in the data segment besides the first data identifier according to the type of the data segment to obtain the source data, the method further includes: parsing the first flag bit to obtain the data format of the data sequence; And / or, After parsing the extended metadata location identifier to obtain the position of the first data fragment in the data sequence, the method further includes: parsing a second flag bit, the second flag bit being an end identifier of the data sequence.

10. A data parsing apparatus, comprising a memory and a processor, characterized in that, The processor is configured to acquire and execute code in the memory to perform the method according to any one of claims 1 to 9.

Images