Data analysis method and device, electronic equipment and storage medium
By employing a multi-layered distributed parsing method in autonomous driving data parsing, and utilizing channels and thread pool queues to process data, the problem of low parsing efficiency under large data volumes is solved, achieving efficient data parsing and algorithm iteration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-28
- Publication Date
- 2026-03-17
AI Technical Summary
When dealing with large amounts of data in autonomous driving, existing technologies suffer from low data parsing efficiency, which affects the efficiency of algorithm iteration.
The target channel is determined from multiple channels, and the data to be parsed is passed to the target thread pool for multi-layer distributed parsing. The thread pool and parsing queue corresponding to each channel are used for data parsing.
It significantly improved data parsing efficiency, enhanced problem analysis and localization efficiency, and facilitated the rapid iteration of autonomous driving algorithms.
Smart Images
Figure CN116302437B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a data parsing method, apparatus, electronic device and storage medium. Background Technology
[0002] Currently, autonomous driving is a highly complex scientific technology. Its research and development process requires continuous testing and iteration, and a series of toolchains exist to support algorithm problem localization, back-injection, and iteration. In problem localization, offline tools are often used to load and analyze recorded data, which includes output data associated with algorithm modules, intermediate variables, and output results. When the data scale is large and there are many problems, the efficiency of developers in analyzing and locating problems is positively correlated with the efficiency of loading and parsing data. Therefore, accelerating data parsing efficiency can effectively promote the iteration of autonomous driving algorithms.
[0003] Therefore, how to improve the data parsing efficiency when dealing with large amounts of data is a technical problem that this invention urgently needs to solve. Summary of the Invention
[0004] This invention provides a data parsing method, apparatus, electronic device, and storage medium, aiming to improve data parsing efficiency.
[0005] The first aspect of this invention provides a data parsing method, the method comprising:
[0006] Among multiple channels, determine the target channel corresponding to the data to be parsed;
[0007] The data to be parsed is transmitted to the target thread pool corresponding to the target channel through the target channel; wherein, each channel corresponds to a separate thread pool, and each thread pool contains a parsing queue corresponding to parsing functions for multiple data types;
[0008] Based on the target data type of the data to be parsed, determine the target parsing function corresponding to the target data type;
[0009] The data to be parsed is passed to the target parsing queue corresponding to the target parsing function for data parsing.
[0010] Optionally, determining the target channel corresponding to the data to be parsed among multiple channels includes:
[0011] Based on the keywords in the data name of the data to be parsed, the data to be parsed is classified to obtain the classification result;
[0012] Based on the classification results, determine the channel corresponding to the classification results among the plurality of channels;
[0013] The channel corresponding to the classification result is determined as the target channel.
[0014] Optionally, the step of passing the data to be parsed into the target parsing queue corresponding to the target parsing function for data parsing includes:
[0015] If data that has been parsed is removed from the target parsing queue, the data to be parsed is passed to the target parsing queue corresponding to the target parsing function for parsing.
[0016] Optionally, the method further includes:
[0017] Obtain the raw data and load it into the cache;
[0018] The original data is filtered based on its data name to determine the data to be parsed.
[0019] Optionally, the data to be parsed is Protobuf data.
[0020] A second aspect of the present invention provides a data parsing apparatus, the apparatus comprising:
[0021] The first determining module is used to determine the target channel corresponding to the data to be parsed among multiple channels;
[0022] The transmission module is used to transmit the data to be parsed to the target thread pool corresponding to the target channel through the target channel; wherein, each channel corresponds to a separate thread pool, and each thread pool contains a parsing queue corresponding to parsing functions for multiple data types;
[0023] The second determining module is used to determine the target parsing function corresponding to the target data type based on the target data type of the data to be parsed;
[0024] The parsing module is used to pass the data to be parsed into the target parsing queue corresponding to the target parsing function for data parsing.
[0025] Optionally, the first determining module includes:
[0026] The classification module is used to classify the data to be parsed based on the keywords in the data name of the data to be parsed, and obtain the classification result;
[0027] The channel determination module is used to determine the channel corresponding to the classification result among the plurality of channels based on the classification result;
[0028] The first determining submodule is used to determine the channel corresponding to the classification result as the target channel.
[0029] Optionally, the parsing module includes:
[0030] The parsing submodule is used to pass the data to be parsed into the target parsing queue corresponding to the target parsing function for parsing when there is data that has been parsed and has left the target parsing queue.
[0031] Optionally, the device further includes:
[0032] The acquisition module is used to acquire raw data and load the raw data into the cache;
[0033] The filtering module is used to filter the original data according to the data name of the original data to determine the data to be parsed.
[0034] Optionally, the data to be parsed is Protobuf data. 。
[0035] A third aspect of the present invention provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the data parsing method of the first aspect of the present invention.
[0036] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data parsing method of the first aspect of the present invention.
[0037] The data parsing method provided in this embodiment of the invention determines the target channel corresponding to the data to be parsed from multiple channels; the data to be parsed is then passed to the target thread pool corresponding to the target channel; each channel corresponds to a separate thread pool, and each thread pool contains parsing queues corresponding to parsing functions for multiple data types; based on the target data type of the data to be parsed, the target parsing function corresponding to the target data type is determined; the data to be parsed is then passed to the target parsing queue corresponding to the target parsing function for data parsing. In this embodiment, for the data parsing of each piece of data to be parsed, a multi-layered distributed method is used to determine the corresponding target thread pool and the target parsing queue within the target thread pool. This allows for simultaneous data parsing of the data to be parsed through multiple independent thread pools and their respective parsing queues, even when the amount of data to be parsed is large, thereby significantly improving the parsing efficiency. Attached Figure Description
[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 This is a flowchart illustrating a data parsing method according to an embodiment of the present invention;
[0040] Figure 2 This invention provides a method for accelerating Protobuf data parsing, as proposed in one embodiment.
[0041] Figure 3 This is a structural block diagram of a data parsing device provided in an embodiment of the present invention;
[0042] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0044] refer to Figure 1 , Figure 1 This is a flowchart illustrating a data parsing method according to an embodiment of the present invention. Figure 1 As shown, the data parsing method in this embodiment may include the following steps:
[0045] Step S11: Among multiple channels, determine the target channel corresponding to the data to be parsed.
[0046] In this embodiment, multiple channels are pre-set, each channel corresponds to a separate thread pool, and each channel is used to transmit the data to be parsed, passing the data to be parsed corresponding to the channel into the thread pool corresponding to that channel.
[0047] Based on this, after obtaining the data to be parsed, this embodiment can determine the channel corresponding to the data from a set of pre-defined channels and designate the channel corresponding to the data as the target channel. In this embodiment, the data to be parsed is the data that needs to be parsed. In one embodiment, the data to be parsed can be relevant data recorded during the autonomous driving research and development process, such as output data associated with various modules, intermediate variables, output results, etc.
[0048] Step S12: The data to be parsed is transmitted to the target thread pool corresponding to the target channel through the target channel; wherein, each channel corresponds to a separate thread pool, and each thread pool contains a parsing queue corresponding to parsing functions for multiple data types.
[0049] In this embodiment, after determining the target channel corresponding to the data to be parsed, the data can be passed to the target thread pool corresponding to the target channel through that target channel. Each channel in this embodiment corresponds to a separate thread pool used for data parsing. Each thread pool stores multiple parsing queues, and each parsing queue is used to parse the data to be parsed according to the data type corresponding to the parsing function. In this embodiment, the target thread pool is the thread pool connected to the target channel.
[0050] Furthermore, the number of parsing queues in each thread pool can be the same or different, and the data type of the parsing function corresponding to the parsing queue in each thread pool can also be the same or different. These can be freely set according to actual scenarios and requirements, and this embodiment does not impose any restrictions on them.
[0051] Step S13: Determine the target parsing function corresponding to the target data type based on the target data type of the data to be parsed.
[0052] In this embodiment, after the function to be parsed enters its corresponding target thread pool, the data type of the function can be determined. Since each channel and its corresponding thread pool may contain data of different data types to be parsed, such as lane line data, map data, and point cloud data, in an optional implementation, the data name of the data to be parsed can be determined first, and then the data type can be determined based on the data type keyword in the data name. For example, the data type keyword in the data name of data A can be used to determine that data A is lane line data, and the data type keyword in the data name of data C can be used to determine that data C is point cloud data, and so on.
[0053] After determining the data type of the function to be parsed, this data type can be designated as the target data type. Then, based on the target data type, the target parsing function corresponding to the target data type is determined from among the multiple parsing queues in the target thread pool. In this embodiment, the target parsing function is either the parsing function corresponding to the data type of the function to be parsed, or it can be the parsing function that performs data parsing on the data to be parsed.
[0054] Step S14: Pass the data to be parsed into the target parsing queue corresponding to the target parsing function for data parsing.
[0055] In this embodiment, after determining the target parsing function of the data to be parsed, the parsing queue corresponding to the target parsing function can be determined, and the parsing queue corresponding to the target parsing function can be determined as the target parsing queue. The data to be parsed is then passed into the target parsing queue for data parsing, thereby completing the data parsing of the data to be parsed.
[0056] In this embodiment, for the data to be parsed, not only is the data not divided into different thread pools through different channels for data parsing, but also, within the same thread pool, it is divided into different parsing queues according to the different data types of the data to be parsed, and the data is parsed simultaneously. This allows the data to be parsed to be parsed simultaneously using multiple thread pools and multiple parsing queues, thereby realizing multi-layered distributed data parsing of the data to be parsed. This greatly improves the data parsing efficiency, enabling developers to significantly improve efficiency in problem analysis and problem localization, and to complete algorithm iterations more quickly, thus greatly supporting the development of autonomous driving algorithms.
[0057] In conjunction with the above embodiments, in one implementation, the present invention also provides a data parsing method. In this method, step S11 specifically includes the following steps:
[0058] Step A: Classify the data to be parsed according to the keywords in the data name to obtain the classification result.
[0059] In this embodiment, the data to be parsed may be classified, and corresponding target channels may be assigned to the data based on the classification results. Specifically, the data names of the data to be parsed may be determined, and the data to be parsed may be distinguished based on the keywords in the data names, thereby obtaining the classification results of the data to be parsed.
[0060] For example, the data to be parsed can be classified according to the module name in the data name, where the module name indicates which module the data to be parsed belongs to. For example, if the module name is A, it means that the data to be parsed is data in module A, so the data in module A is classified into one category; if the module name is D, it means that the data to be parsed is data in module D, so the data in module D is classified into another category, and so on.
[0061] Step B: Based on the classification results, determine the channel corresponding to the classification results among the multiple channels.
[0062] In this embodiment, multiple channels are pre-established based on the classification results, meaning each classification result corresponds to a specific channel. Therefore, based on the classification result of the data to be parsed, the channel corresponding to that classification result is determined from among the multiple channels.
[0063] Step C: Determine the channel corresponding to the classification result as the target channel.
[0064] In this embodiment, after determining the channel corresponding to the classification result of the data to be parsed, the channel corresponding to the classification result of the data to be parsed can be determined as the target channel of the data to be parsed, so that the data to be parsed can be passed into the corresponding target thread pool according to the target channel.
[0065] Understandably, for multiple pieces of data to be parsed, the corresponding target channels can be determined based on the classification results of the multiple pieces of data to be parsed, and then the multiple pieces of data to be parsed can be passed into their respective target thread pools, thereby completing the first-level distributed parsing of the data to be parsed and initially improving the parsing efficiency of the data to be parsed.
[0066] In this embodiment, it is considered that if the parsing of a certain piece of data in each thread pool takes a long time, other pieces of data will have to wait for the parsing of that long-running data to be completed before the parsing operation can be performed. When the data volume is large, this will cause the parsing queue to be blocked, resulting in long processing times and extremely low efficiency. Therefore, based on this, this embodiment performs two-level distributed parsing for the data to be parsed in each thread pool, so that each piece of data to be parsed can be parsed through a different parsing queue according to its data type, thereby further improving the parsing efficiency of the data to be parsed.
[0067] In conjunction with the above embodiments, in one implementation, the present invention also provides an image processing method. In this method, step S14 specifically includes the following steps:
[0068] Step D: If data that has been parsed has exited the target parsing queue, the data to be parsed is passed to the target parsing queue corresponding to the target parsing function for parsing.
[0069] In this embodiment, when the target parsing function of the data to be parsed is determined, the target parsing queue corresponding to the target parsing function can be monitored. When it is detected that data that has been parsed has left the target parsing queue, the data to be parsed is passed into the target parsing queue corresponding to the target parsing function, so as to perform data parsing.
[0070] It is understandable that the parsing queue may contain multiple pieces of data waiting to be parsed. The parsing queue is a queue where data is parsed sequentially, with the next piece of data only entering the queue after the previous piece of data has been parsed and removed from the queue. In this embodiment, the parsed data refers to the parsing result of the data to be parsed. The parsing result can refer to the parsed data obtained after the data to be parsed was successfully parsed, or it can refer to the null value (i.e., no return value) obtained when the data to be parsed could not be parsed.
[0071] In conjunction with the above embodiments, in one implementation, the present invention also provides a data parsing method. In addition to the steps described above, this method may further include the following steps:
[0072] Step E: Obtain the raw data and load it into the cache.
[0073] In this embodiment, the raw data can be obtained first. The raw data consists of all data generated and recorded during the project development process, and then the obtained raw data is loaded into the cache. Specifically, in this embodiment, after retrieving the raw data from the database, the retrieved raw data needs to be stored in the local cache.
[0074] Step F: Filter the original data according to the data name of the original data to determine the data to be parsed.
[0075] In this embodiment, after the original data is cached and loaded, the original data can be filtered according to its data name, thereby determining the data to be parsed from the original data.
[0076] In this embodiment, keyword tracking can be implemented in the data name beforehand, such as tracking the module keyword in the data name. After loading the original data, the data name of the original data is obtained, and then the tracking points in the data name are used to determine whether each piece of original data needs to be parsed. If the keyword in the data name matches the tracking point, that original data is identified as data to be parsed. For example, however, if data parsing of the SWC module is required in this case, tracking the keyword "SWC" will ensure that all data in the original data name containing the module name "SWC" is selected as data to be parsed.
[0077] In conjunction with the above embodiments, in one implementation, this invention also provides a data parsing method. In this method, the data to be parsed is Protobuf data, such as data related to Protobuf format storage in an autonomous driving system.
[0078] In the data recording module for autonomous driving development, the vast majority of the data stored is Protobuf data. Since Protobuf data based on the Protobuf protocol involves serialization and deserialization when parsing each frame of Protobuf data, the time consumption of this part of the parsing is unavoidable. Therefore, for Protobuf data, the data parsing method of any of the above embodiments can also be used to focus the main work on how to put the Protobuf data of different frames into different parsers, and to place the data as reasonably and efficiently as possible, thereby greatly improving the data parsing efficiency.
[0079] like Figure 2 As shown, Figure 2 This is a method for accelerating Protobuf data parsing proposed in one embodiment of the present invention. Figure 2 The first step is to load the data. In this embodiment, the data is Protobuf data, so the Protobuf type data is recorded into the cache.
[0080] Then, data filtering is performed on the Protobuf data. The need for parsing is determined by the data name of each frame of the Protobuf data, thus identifying the data to be parsed. Specifically, this is done by pre-implementing tracking points for key keywords in the data names, allowing the filtering of data to be parsed based on these tracking points. For example, if the Protobuf data name is SWC-Package.Message-XXX, where SWC is the algorithm module name, Package.Message indicates the message from the proto module used, and XXX is a user-defined suffix, and the pre-implemented tracking point is the keyword "SWC," then data with names matching "SWC" are considered data to be parsed.
[0081] Next, the corresponding channel is determined based on the data name. In this embodiment, multiple channels are pre-established, each with a separate thread pool backend, ensuring they do not interfere with each other. Each channel has a pre-defined classification result for the data to be parsed, allowing the corresponding channel to be determined based on this classification. For example, the data could be categorized into target data, map data, point cloud data, and lane line data channels, thus assigning each piece of data to different thread pools for parsing.
[0082] Each thread pool has a parsing queue with multiple parsing functions for different data types. For data entering each thread pool, the target parsing function is determined based on the data type, and then the target parsing queue is determined based on the target parsing function. The data is then passed to its corresponding target parsing queue for parsing. In this way, each thread pool has multiple parsing queues for data parsing, further reducing the queuing time for parsing data and improving data parsing efficiency through a two-layer distributed approach.
[0083] Finally, the parsed data is summarized and output, for example, through visualization, thus completing the data parsing process.
[0084] This embodiment proposes a method to accelerate Protobuf data parsing, which is data-driven in autonomous driving. This method is accurate and effective in project implementation, greatly improving the efficiency of developers in problem analysis and localization, enabling them to complete algorithm iterations more quickly, and providing significant support for the development of autonomous driving algorithms.
[0085] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.
[0086] Based on the same inventive concept, one embodiment of the present invention provides a data parsing device 300. (See reference...) Figure 3 , Figure 3 This is a structural block diagram of a data parsing device provided in an embodiment of the present invention. Figure 3 As shown, the device 300 includes:
[0087] The first determining module 301 is used to determine the target channel corresponding to the data to be parsed among multiple channels;
[0088] The transmission module 302 is used to transmit the data to be parsed to the target thread pool corresponding to the target channel through the target channel; wherein, each channel corresponds to a separate thread pool, and each thread pool contains a parsing queue corresponding to parsing functions for multiple data types;
[0089] The second determining module 303 is used to determine the target parsing function corresponding to the target data type based on the target data type of the data to be parsed;
[0090] The parsing module 304 is used to pass the data to be parsed into the target parsing queue corresponding to the target parsing function for data parsing.
[0091] Optionally, the first determining module 301 includes:
[0092] The classification module is used to classify the data to be parsed based on the keywords in the data name of the data to be parsed, and obtain the classification result;
[0093] The channel determination module is used to determine the channel corresponding to the classification result among the plurality of channels based on the classification result;
[0094] The first determining submodule is used to determine the channel corresponding to the classification result as the target channel.
[0095] Optionally, the parsing module 304 includes:
[0096] The parsing submodule is used to pass the data to be parsed into the target parsing queue corresponding to the target parsing function for parsing when there is data that has been parsed and has left the target parsing queue.
[0097] Optionally, the device 300 further includes:
[0098] The acquisition module is used to acquire raw data and load the raw data into the cache;
[0099] The filtering module is used to filter the original data according to the data name of the original data to determine the data to be parsed.
[0100] Optionally, the data to be parsed is Protobuf data. 。
[0101] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the data parsing method as described in any of the above embodiments of the present invention.
[0102] Based on the same inventive concept, another embodiment of the present invention provides an electronic device 400, such as... Figure 4 As shown. Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory 402, a processor 401, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps of the data parsing method described in any of the above embodiments of the present invention.
[0103] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0104] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0105] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0106] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0109] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.
[0110] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0111] The data parsing method, apparatus, electronic device, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A data parsing method, characterized by, The method is applied to the field of automatic driving, and the method comprises: In a plurality of channels, a target channel corresponding to to-be-analyzed data is determined; the to-be-analyzed data is determined by performing keyword implantation screening based on a data name on original data; The to-be-analyzed data is transmitted into a target thread pool corresponding to the target channel through the target channel; wherein each channel corresponds to a separate thread pool, and in each thread pool, there are a plurality of data type analysis function corresponding analysis queues; the analysis queues in the same thread pool are divided according to the data type of the to-be-analyzed data, and each analysis queue corresponds to an analysis function of one data type; A target data type of the to-be-analyzed data is determined based on a data type keyword in the data name of the to-be-analyzed data; A target analysis function corresponding to the target data type is determined according to the target data type of the to-be-analyzed data; The to-be-analyzed data is transmitted into a target analysis queue corresponding to the target analysis function for data analysis.
2. The data parsing method of claim 1, wherein, The target channel corresponding to the to-be-analyzed data is determined in a plurality of channels, comprising: The to-be-analyzed data is classified according to a keyword in the data name of the to-be-analyzed data to obtain a classification result; According to the classification result, a channel corresponding to the classification result is determined in the plurality of channels; The channel corresponding to the classification result is determined as the target channel.
3. The data parsing method of claim 1, wherein, The to-be-analyzed data is transmitted into a target analysis queue corresponding to the target analysis function for data analysis, comprising: In the case that there is analyzed data exiting the target analysis queue, the to-be-analyzed data is transmitted into the target analysis queue corresponding to the target analysis function for analysis.
4. The data parsing method of claim 1, wherein, The method further comprises: Original data is obtained, and the original data is loaded into a cache; The original data is screened according to a data name of the original data to determine the to-be-analyzed data.
5. The data parsing method according to any one of claims 1 to 4, characterized in that, The to-be-analyzed data is Protobuf data.
6. A data parsing device, characterized by, The device is applied to the field of automatic driving, and the device comprises: A first determination module is configured to determine, in a plurality of channels, a target channel corresponding to to-be-analyzed data; the to-be-analyzed data is determined by performing keyword implantation screening based on a data name on original data; A transmission module is configured to transmit the to-be-analyzed data into a target thread pool corresponding to the target channel through the target channel; wherein each channel corresponds to a separate thread pool, and in each thread pool, there are a plurality of data type analysis function corresponding analysis queues; the analysis queues in the same thread pool are divided according to the data type of the to-be-analyzed data, and each analysis queue corresponds to an analysis function of one data type; A second determination module is configured to determine, according to a target data type of the to-be-analyzed data, a target analysis function corresponding to the target data type; the target data type of the to-be-analyzed data is determined based on a data type keyword in the data name of the to-be-analyzed data; An analysis module is configured to transmit the to-be-analyzed data into a target analysis queue corresponding to the target analysis function for data analysis.
7. The data parsing device according to claim 6, wherein The first determination module comprises: The classification module is configured to classify the data to be parsed according to a keyword in a data name of the data to be parsed, and obtain a classification result. The channel determination module is configured to determine, according to the classification result, a channel corresponding to the classification result from the multiple channels. The first determination sub-module is configured to determine the channel corresponding to the classification result as a target channel.
8. The data parsing device of claim 6, wherein, The analysis module comprises: The analysis sub-module is configured to, in a case where there is data that has completed analysis exiting the target analysis queue, transmit the data to be parsed into a target analysis queue corresponding to the target analysis function for analysis.
9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the data analysis method according to any one of claims 1 to 5.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the data analysis method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Distributed storage system request processing method and device, equipment and medium
CN111309478A
Log analysis method and device, computing equipment and storage medium
CN114968958A