A data parsing method and device based on a syntax analysis generator

Abstract

The application discloses a data analysis method and device based on a syntax analysis generator, and relates to the technical field of data processing. Byte stream data to be analyzed in a task scene is acquired; a CSG syntax file is created according to a syntax specification of the syntax analysis generator, and context-related syntax rules of a protocol are defined; the CSG syntax file is processed by using the syntax analysis generator, abstract syntax tree information related to the CSG syntax file is generated, and a Java parser is generated based on the abstract syntax tree information; a CsgVisitor class is realized based on the generated Java parser, the CSG syntax file is parsed, and a Protocol protocol object is constructed; a CSGPVisitor accessor and a CSGPParser parser are created, the byte stream data is parsed in combination of the Protocol protocol object and preset analysis rules.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a data analysis method and device based on a syntax analysis generator. BACKGROUND

[0002] Traditional interface description languages (such as IDL, WSDL, etc.) usually adopt static syntax definitions and cannot flexibly adapt to changes in syntax rules in different contexts.

[0003] In typical network protocols (such as HTTP protocols), syntax rules often depend on dynamic changes in runtime contexts, and different request methods may require different parameter structures and rules. Such scenarios require parsing logic to be dynamically adjusted according to contexts, but traditional IDL tools (such as Thrift, Protocol Buffers, etc.) can define data structures and transmit data through built-in transmission protocols, but their syntax definitions are relatively fixed and lack support for context-related syntax rule data. Moreover, most custom application network protocols do not have syntax definition files, and the protocol content cannot be determined through a syntax file, and there is a lack of general context-related grammar parsing implementation. SUMMARY

[0004] In the embodiments of the present application, by providing a data analysis method based on a syntax analysis generator, the problem of lack of support for context-related syntax rule data and lack of general context-related grammar parsing implementation in the prior art is solved.

[0005] In a first aspect, the embodiments of the present application provide a data analysis method based on a syntax analysis generator, which comprises: obtaining byte stream data to be parsed in a task scenario; creating a CSG syntax file according to syntax specifications of a syntax analysis generator, defining context-related syntax rules of a protocol; processing the CSG syntax file by using the syntax analysis generator, generating abstract syntax tree information related to the CSG syntax file, and generating a Java parser based on the abstract syntax tree information; implementing a custom CsgVisitor class based on the generated Java parser, parsing the CSG syntax file and constructing a Protocol protocol object; creating a CSGPVisitor accessor and a CSGPParser parser, combining the Protocol protocol object and a preset parsing rule, and parsing the byte stream data.

[0006] In a possible implementation, the creating a CSG grammar file according to the syntax specification of the syntax analysis generator, and defining context-related syntax rules of the protocol comprises: defining CSG grammar metadata information, including the name, version, and remarks of the grammar, and the syntax definition of the protocol; providing content definition of the protocol, including content column definition, and distinguishing the syntax definition of normal columns and combined columns; defining attributes for the normal columns and the combined columns respectively, the normal column attributes including the ID, name, data type, data length, rendering type, and remarks of the column, and the combined column attributes including the ID, name, rendering type, selection field, and data content of the column; and providing basic lexical and syntax definitions.

[0007] In a possible implementation, the processing the CSG grammar file by using the syntax analysis generator, generating abstract syntax tree information related to the CSG grammar file, and generating a Java parser based on the abstract syntax tree information comprises: taking the CSG grammar file meeting the requirements of the syntax analysis generator as input, running the syntax analysis generator to perform lexical analysis, identifying basic elements of the language and converting the basic elements into a symbol stream; performing semantic analysis by the syntax analysis generator, combining and verifying the basic elements in the symbol stream according to the context-related syntax rules, generating the abstract syntax tree information related to the CSG grammar file, and generating the Java parser based on the abstract syntax tree information; and calling a cli command in the syntax analysis generator to generate basic auxiliary data and auxiliary classes to jointly process the CSG grammar file with the Java parser.

[0008] In a possible implementation, the generating the Java parser based on the generated Java parser, implementing a custom CsgVisitor class, parsing the CSG grammar file, and constructing a Protocol protocol object comprises: parsing the abstract syntax tree of the CSG grammar file by using the Java parser; defining the CsgVisitor class, which inherits a CsgBaseVisitor class generated by the syntax analysis generator; and the CsgVisitor class is used to recursively call a visit method to traverse the abstract syntax tree of the CSG grammar file from a child node to a root node, and construct the Protocol protocol object; and the child node returns a basic object, and the root node aggregates results of the child nodes.

[0009] In a possible implementation, the creating the CSGPVisitor accessor and the CSGPParser parser, in combination with the Protocol protocol object and preset parsing rules, performs parsing on the byte stream data, including: creating a CSGPVisitor accessor interface to define processing logic for different types of columns and the whole protocol in the parsing process; creating a CSGPParser parser to recursively access column definitions of the Protocol protocol object, buffering column data in the first access, obtaining data from a cache object when a length reference column is encountered, and returning column information and parsed data to a user through the CSGPVisitor accessor interface; the CSGPParser parser integrates a function of the CsgVisitor class, and uses the CSGPVisitor accessor to implement user access logic for parsed data, in combination with the Protocol protocol object and preset parsing rules, to perform parsing on the byte stream data.

[0010] In a second aspect, the embodiments of the present application provide a data parsing device based on a syntax analysis generator, which includes: an obtaining module, configured to obtain byte stream data to be parsed in a task scenario; a defining module, configured to create a CSG syntax file according to a syntax specification of the syntax analysis generator, and define context-related syntax rules of a protocol; a generating module, configured to process the CSG syntax file by using the syntax analysis generator, generate abstract syntax tree information related to the CSG syntax file, and generate a Java parser based on the abstract syntax tree information; a parsing module, configured to implement a custom CsgVisitor class based on the generated Java parser, parse the CSG syntax file, and build a Protocol protocol object; and a creating module, configured to create a CSGPVisitor accessor and a CSGPParser parser, in combination with the Protocol protocol object and preset parsing rules, to perform parsing on the byte stream data.

[0011] In a third aspect, the embodiments of the present application provide a data parsing server based on a syntax analysis generator, including a memory and a processor; the memory is configured to store computer executable instructions; and the processor is configured to execute the computer executable instructions to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0012] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores executable instructions, and a computer executes the executable instructions to implement the method in the first aspect or any possible implementation manner of the first aspect.

[0013] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects:

[0014] The embodiment of the present application provides a data analysis method based on a syntax analysis generator, acquires byte stream data to be analyzed in a task scene. A CSG syntax file is created according to a syntax specification of the syntax analysis generator, context-related syntax rules of a protocol are defined, dynamic binding of syntax rules and runtime contexts is realized, and the limitation that an existing IDL tool cannot express conditional analysis is solved. The CSG syntax file is processed by using the syntax analysis generator, abstract syntax tree information related to the CSG syntax file is generated, and a Java parser is generated based on the abstract syntax tree information. According to the present application, related syntax rule modification only needs to adjust the CSG file, and the code of the Java parser does not need to be changed. Based on the generated Java parser, a custom CsgVisitor class is realized, the CSG syntax file is parsed and a Protocol protocol object is constructed. A CSGPVisitor accessor and a CSGPParser parser are created, the byte stream data is parsed by combining the Protocol protocol object and preset analysis rules. The problems that the prior art lacks support for context-related syntax rule data and lacks a general context-related grammar analysis implementation are solved. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the prior art. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0016] Figure 1 A flowchart of a data analysis method based on a syntax analysis generator provided by the embodiment of the present application is provided.

[0017] Figure 2 A schematic diagram of a data analysis device based on a syntax analysis generator provided by the embodiment of the present application is provided.

[0018] Figure 3 A schematic diagram of a data analysis server based on a syntax analysis generator provided by the embodiment of the present application is provided. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0020] The following descriptions of some of the technologies involved in the embodiments of the present application are provided to facilitate understanding. They should be considered only as exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the sake of clarity and conciseness, the following description omits some descriptions of well-known functions and structures.

[0021] The embodiments of the present application provide a data parsing method based on a grammar analysis generator, as shown in Figure 1 The method comprises steps S101 to S105. In which, Figure 1 The embodiments of the present application only show one execution order, and do not represent the only execution order of the data parsing method based on the grammar analysis generator. As long as the final result can be achieved, Figure 1 The steps shown can be executed in parallel or in reverse.

[0022] S101: Obtain byte stream data to be parsed in a task scenario.

[0023] Specifically, the task scenario of the present application can be network communication, file storage, message queue, etc. When the task scenario is network communication, the task requirement is to parse binary data in HTTP request body (such as JSON data submitted by the client in Web API) or custom TCP protocol (such as binary instructions between the server and the client). When the task scenario is file storage, the task requirement is to parse local binary files (such as logs, database files). When the task scenario is a message queue, the task requirement is to parse binary messages received from the message queue.

[0024] S102: Create a CSG grammar file according to the grammar specification of the grammar analysis generator, and define the context-related grammar rules of the protocol.

[0025] Creating a CSG grammar file according to the grammar specification of the grammar analysis generator, and defining the context-related grammar rules of the protocol, includes the following contents.

[0026] Specifically, the grammar analysis generator of the present application can be Antlr4. The CSG (Context Sensitive Grammar) grammar file has a.g4 extension, and defines the context-related rules of the protocol.

[0027] Define the CSG grammar metadata information, including the name, version, remarks of the grammar, and the grammar definition of the protocol.

[0028] Specifically, the metadata is used to identify the basic information of the protocol. The name is the unique identifier of the protocol. The version is the protocol version number, which supports version compatibility verification. The remarks are the purpose of the protocol. The grammar definition of the protocol is the root rule of the protocol, which contains data column definition.

[0029] The content definition of the protocol is provided, including content column definition, syntax definition distinguishing normal column and combined column.

[0030] Specifically, the content definition of the protocol is used to describe the data structure of the protocol, supporting both normal column and combined column. The normal column is a fixed-length data field (such as integer, string). The combined column is a nested structure containing sub-columns (such as an array in a JSON object).

[0031] The attribute definition is made respectively for normal column and combined column. The normal column attribute includes the ID, name, data type, data length, rendering type and note of the column. The combined column attribute includes the ID, name, rendering type, selection field and data content of the column.

[0032] Specifically, the data length of the normal column attribute adds the data definition of field reference in the definition, realizing the parsing mechanism of context-dependent grammar. In the traditional data length definition, a fixed value is usually simply specified as the length. However, in the CSG grammar of the present application, in order to realize the parsing mechanism of context-dependent grammar, the data definition of field reference is introduced. For example, in a data structure containing multiple columns, the data length of a certain column may depend on the value of another column. For example, in a data parsing scene of a network protocol, there is a column A representing the length of data block, and another column B representing the data content. The data length of column B can refer to the value of column A. The above process can be realized by column_data_length: integer׀'ref''(''string'')'. Wherein, column_data_length is the data length of the column, integer is the fixed length, ref''(''string'')' is the reference length, and string is the field name.

[0033] In the parsing process, when a column using field reference to define data length is encountered, the parser will first find the referenced column. If the referenced column has not been parsed, the referenced column will be parsed first to obtain its value, and then the value is taken as the data length of the current column.

[0034] Basic lexical and syntax definitions are provided.

[0035] In particular, in the protocol definition, there are various different types of data attributes, such as string, byte, etc. Accurate lexical and syntax definitions of these simple attributes enable the Java parser to correctly identify and process the occurrence of these data types in the protocol description. For example, in defining a protocol containing a device name (string type) and a device identification (byte type), the Java parser needs to know how to extract these information from the input syntax description.

[0036] S103: processing the CSG syntax file by using the syntax analysis generator to generate abstract syntax tree information related to the CSG syntax file, and generating the Java parser based on the abstract syntax tree information.

[0037] Processing the CSG syntax file by using the syntax analysis generator to generate abstract syntax tree information related to the CSG syntax file, and generating the Java parser based on the abstract syntax tree information, includes the following.

[0038] The CSG syntax file meeting the requirements of the syntax analysis generator is taken as input, and the syntax analysis generator is run for lexical analysis to identify the basic elements of the language and convert them into a symbol stream.

[0039] In particular, the CSG syntax file meeting the requirements of the syntax analysis generator is given an extension of.g4. Running the syntax analysis generator, it first performs lexical analysis on the input CSG syntax file. The main task of lexical analysis is to identify the basic elements of the language, such as keywords, identifiers, operators, constants, etc., and convert them into a symbol stream (token stream). For example, the keywords defined in the CSG syntax file, such as version (used to specify the version information of the protocol), name (to specify the name of the protocol), byte, etc., and the identifiers such as column name, ID, etc. defined by the user are identified and assigned to the corresponding symbol type.

[0040] The syntax analysis generator performs semantic analysis, combines and verifies the basic elements in the symbol stream according to the context-dependent syntax rules, generates abstract syntax tree information related to the CSG syntax file, and generates the Java parser based on the abstract syntax tree information.

[0041] Specifically, the purpose of semantic analysis is to combine and verify the basic elements in the symbol stream according to the context-related syntax rules. It checks the correctness of the syntax structure and ensures that the combination of symbols complies with the defined syntax rules. For example, when defining the content of a protocol, it checks whether the definitions of ordinary data blocks and combined data blocks conform to the specifications, whether the definitions of column attributes are complete and reasonable, etc. After semantic analysis, the syntax analysis generator generates abstract syntax tree information related to the CSG syntax file. The abstract syntax tree is a tree structure that represents the hierarchical structure and logical relationship of the syntax rules in a more concise and abstract way. For example, the definition of a protocol is the root node, the metadata information, content definition, etc. under it are the child nodes, and the column attribute definition is further the lower layer node. Through the abstract syntax tree, the organization of the syntax rules can be clearly seen, providing structured information for generating Java parsers.

[0042] Further, the syntax analysis generator automatically generates the parser code of the target language (Java in this application) according to the generated abstract syntax tree information. It will generate corresponding Java classes and methods according to the structure of the abstract syntax tree and the syntax rules, and these codes can realize the conversion of the input text that complies with the CSG syntax rules into the computer internal data structure. For example, Lexer class for lexical analysis and Parser class for syntax analysis, as well as related auxiliary classes and methods are generated. The generated Java parser has the ability to parse the description text in the input CSG syntax file into structured data. It can recognize various syntax elements in the text and verify and process them according to the defined rules. For example, when an input protocol definition text that complies with the CSG syntax rules is input, the Java parser can parse it into a protocol object model in memory, providing a basis for subsequent data parsing.

[0043] Call the cli command in the syntax analysis generator to generate basic auxiliary data and auxiliary classes to work with the Java parser to process the CSG syntax file.

[0044] Specifically, the cli command can be java-jar antlr-4.13.2-complete.jar -listener -visitor -o src\main\java\grammar\csg src\main\antlr4\grammar\Csg.g4. Wherein, java-jar antlr-4.13.2-complete.jar is used to execute the generation command of the syntax analyzer generator. -listener is used to generate the Listener mode of the accessor of the CSG syntax file. -visitor is used to generate the Visitor mode of the accessor of the CSG syntax file. -o is used to store the generated auxiliary data and auxiliary classes. src\main\antlr4\grammar\Csg.g4 is the syntax definition file of the syntax. The basic auxiliary data and auxiliary classes are generated. These auxiliary data and classes cooperate with the generated Java parser to process the CSG syntax file. For example, the generated Listener and Visitor interfaces and related implementation classes can help users traverse and process the parsed abstract syntax tree in different ways. The Listener interface provides an event-driven way. When the parser traverses different nodes of the abstract syntax tree, corresponding events are triggered, and users can respond to these events by implementing the methods of the Listener interface. The Visitor interface provides an active way to traverse the abstract syntax tree. Users can implement the Visitor interface to access and process each node of the abstract syntax tree according to their own needs.

[0045] S104: Based on the generated Java parser, implement a custom CsgVisitor class to parse the CSG syntax file and build a Protocol protocol object.

[0046] Based on the generated Java parser, implement a custom CsgVisitor class to parse the CSG syntax file and build a Protocol protocol object, including the following.

[0047] Use the Java parser to parse the abstract syntax tree of the CSG syntax file.

[0048] Specifically, the abstract syntax tree represents the syntax structure of the CSG syntax file in a tree structure.

[0049] Define the CsgVisitor class, which inherits the CsgBaseVisitor class generated by the syntax analyzer generator.

[0050] Specifically, the CsgBaseVisitor class is a base class generated by the grammar analyzer according to the CSG grammar file, which provides a basic framework and methods for visiting each node of the abstract syntax tree. By inheriting the CsgBaseVisitor class, the custom CsgVisitor class can override these methods to implement specific parsing logic. For example, methods for visiting different syntax rule nodes have been defined in the CsgBaseVisitor class, and the CsgVisitor class can implement these methods according to the specific meaning of the CSG grammar to extract and process the information of the nodes.

[0051] The CsgVisitor class is used to recursively call the visit method to traverse the abstract syntax tree of the CSG grammar file from the child node to the root node, and build the Protocol protocol object; wherein the child node returns the base object, and the root node aggregates the child node results.

[0052] Specifically, the visit method in the CsgVisitor class will recursively call according to the type of the abstract syntax tree node. Starting from the leaf node, each visit method will process the corresponding syntax element and return a base object containing the information parsed from the node. With the recursion, the information of these base objects will be gradually aggregated upwards, and finally the complete Protocol protocol object will be built in the visit method of the root node. For example, when a child node representing column definition is traversed, the corresponding visit method will extract the column's ID, name, data type, etc. information and encapsulate it into a column object (such as SimpleColumn or CombinedColumn). With the recursion rising, these column objects will be aggregated into the parent node's corresponding object, such as adding column objects to a Definition object representing protocol data definition (used to encapsulate metadata or structure information). When the recursion reaches the root node of the abstract syntax tree, the visit method of the root node will integrate all the information aggregated by the child nodes to build the final Protocol protocol object, which contains the version, name, description, and detailed data definition of the protocol, etc.

[0053] S105: Create a CSGPVisitor accessor and a CSGPParser parser, combine the Protocol protocol object and the preset parsing rules, and parse the byte stream data.

[0054] Create a CSGPVisitor accessor and a CSGPParser parser, combine the Protocol protocol object and the preset parsing rules, and parse the byte stream data, including the following.

[0055] Create CSGPVisitor accessor interface to define the processing logic of different types of columns and protocol as a whole in the parsing process.

[0056] Specifically, by defining the CSGPVisitor accessor interface, different operations in the parsing process are abstracted, so that users can implement different access logic according to their business needs, thereby flexibly processing the parsed data. The interface usually contains a series of methods for processing different stages and different types of columns in the protocol parsing process. The methods contained in the CSGPVisitor accessor interface are as follows.

[0057] onVisitSimpleColumn(SimpleColumn column, byte[] data), called when a simple column is parsed, the SimpleColumn parameter contains the definition information of the column, and the byte[] data is the actual data of the column parsed from the byte stream. Users can process the simple column data in this method, such as data conversion, verification, etc. onVisitCombinedColumnBegin(CombinedColumn column) and onVisitCombinedColumnEnd(CombinedColumn column) are called when starting to parse a combined column and ending to parse a combined column, respectively, for processing the start and end logic of the combined column, such as initializing the data structure of the combined column, processing the nesting relationship of the combined column, etc. onVisitBegin(Protocol protocol) and onVisitEnd(Protocol protocol) are called when the protocol parsing starts and ends, which can be used for some global initialization and cleaning operations, such as initializing the data storage structure, outputting the parsing result, etc.

[0058] Create CSGPParser parser to recursively access the column definitions of the Protocol protocol object, buffer the column data in the first access, get the data from the cache object when encountering a length reference column, and return the column information and parsed data to the user through the CSGPVisitor accessor interface.

[0059] Specifically, the CSGPParser parser recursively visits and parses the input byte stream data according to the column definitions of the Protocol protocol object. During the parsing process, different types of data blocks, including simple columns and combined columns, are processed according to the preset parsing rules, and the parsed data is returned to the user through the CSGPVisitor accessor interface. The CSGPParser parser starts from the top-level column definition of the Protocol protocol object and recursively visits each column definition. For a simple column, the corresponding length of data is directly read from the byte stream; for a combined column, the starting logic of the combined column is first processed, then the contained sub-column definitions are recursively processed, and finally the ending logic of the combined column is processed. When visiting the column data for the first time, the parser will buffer the parsed data. When encountering a length reference column (i.e., the length of the column depends on the value of another column), the CSGPParser parser will obtain the corresponding data from the cache object to determine the length of the current column, and then read the actual data from the byte stream. During the parsing process, the CSGPParser parser will call the corresponding methods in the CSGPVisitor accessor interface according to different parsing stages and column types, and pass the column information and parsed data to the user. By implementing the CSGPVisitor accessor interface, the user can customize the processing logic for these data, thereby realizing flexible data access.

[0060] The CSGPParser parser integrates the functions of the CsgVisitor class, realizes user access logic for parsed data using the CSGPVisitor accessor, and parses the byte stream data in combination with the Protocol protocol object and the preset parsing rules.

[0061] Specifically, the Protocol protocol object contains detailed definition information of the protocol, such as the ID, name, data type, length, and rendering type of the column. These information provides the basis for data parsing for the CSGPParser parser, and the CSGPParser parser accurately extracts the data of each column from the byte stream according to these definition information.

[0062] The preset parsing rules include data type conversion rules and data length calculation rules. For example, for an integer type column, the parser will convert the binary data in the byte stream to an integer; for a length reference column, the parser will calculate the actual length of the current column according to the value of the referenced column.

[0063] An example of the parsing process is as follows.

[0064] The CSGPParser parser first reads the Protocol protocol object to obtain basic information and column definitions of the protocol. Starting from the beginning of the byte stream, each column is parsed in sequence according to the order of the column definitions. For a simple column, data of a corresponding length is read from the byte stream according to the data type and length of the column, and data conversion is performed according to preset parsing rules. For a combined column, the starting logic of the combined column is processed first, then the sub-column definitions contained in the combined column are processed recursively, and finally the ending logic of the combined column is processed. During the parsing process, the parsed column information and data are returned to the user through the CSGPVisitor accessor interface, and the user obtains the structured business data.

[0065] The embodiment of the application further provides a data parsing device 200 based on a syntax analysis generator, as shown in the figure, the device comprises an acquisition module 201, a definition module 202, a generation module 203, a parsing module 204 and a creation module 205. Figure 2

[0066] The acquisition module 201 is used for acquiring byte stream data to be parsed in a task scenario.

[0067] The definition module 202 is used for creating a CSG syntax file according to a syntax specification of the syntax analysis generator, and defining context-related syntax rules of a protocol.

[0068] The generation module 203 is used for processing the CSG syntax file by using the syntax analysis generator, generating abstract syntax tree information related to the CSG syntax file, and generating a Java parser based on the abstract syntax tree information.

[0069] The parsing module 204 is used for realizing a custom CsgVisitor class based on the generated Java parser, parsing the CSG syntax file and constructing a Protocol protocol object.

[0070] The creation module 205 is used for creating a CSGPVisitor accessor and a CSGPParser parser, and parsing byte stream data in combination with the Protocol protocol object and preset parsing rules.

[0071] Some modules in the device described in the application can be described in the general context of computer-executable instructions, such as program modules, which are executed by computers. Generally, program modules include routines, programs, objects, components, data structures, classes, and the like, which perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0072] ​The apparatuses or modules illustrated in the above application examples can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above apparatuses are described as various modules with functions. In the implementation of the application examples, the functions of the modules can be implemented in one or more software and / or hardware. Of course, the modules with certain functions can also be implemented by a combination of multiple sub-modules or sub-units.

[0073] The methods, apparatuses or modules described in the present application can be implemented in a computer readable program code, and the controller can be implemented in any appropriate manner, for example, the controller can take the form of, for example, a microprocessor or a processor, and a computer readable medium storing computer readable program code (for example, software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, abbreviated as: ASIC), programmable logic controllers and embedded microcontrollers. Examples of the controller include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a pure computer readable program code, the same function can also be implemented by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, such a controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the devices for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0074] As shown in Figure 3 The application examples also provide a data parsing server based on a syntax analysis generator, which comprises a memory 301 and a processor 302; the memory 301 is used to store computer executable instructions; and the processor 302 is used to execute the computer executable instructions to implement the data parsing method based on the syntax analysis generator.

[0075] The application examples also provide a computer readable storage medium, which stores executable instructions, and a computer executing the executable instructions can implement the data parsing method based on the syntax analysis generator.

[0076] From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product or can be embodied in the implementation process of data migration. The computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the embodiments of the present application.

[0077] The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. The whole or part of the present application can be used in a plurality of general or special computer system environments or configurations.

[0078] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the present application.

Claims

1. A data parsing method based on a syntax analyzer generator, characterized in that, include: Obtain the byte stream data to be parsed in the task scenario; Create a CSG syntax file based on the syntax specification of the syntax analyzer generator, and define the context-dependent syntax rules of the protocol; The syntax analyzer generator processes CSG syntax files, generates abstract syntax tree information related to the CSG syntax files, and generates a Java parser based on the abstract syntax tree information. Based on the generated Java parser, implement a custom CsgVisitor class to parse CSG syntax files and construct Protocol objects; Create a CSGPVisitor accessor and a CSGPParser parser, and parse the byte stream data by combining the Protocol object and the preset parsing rules; The process of using a syntax analyzer to process CSG syntax files, generate abstract syntax tree information related to the CSG syntax files, and generate a Java parser based on the abstract syntax tree information includes: taking a CSG syntax file that meets the requirements of the syntax analyzer as input, running the syntax analyzer to perform lexical analysis, identifying the basic elements of the language and converting them into a symbol stream; the syntax analyzer to perform semantic analysis, combining and verifying the basic elements in the symbol stream according to context-dependent grammar rules, generating abstract syntax tree information related to the CSG syntax files, and generating a Java parser based on the abstract syntax tree information; and calling CLI commands in the syntax analyzer to generate basic auxiliary data and auxiliary classes to process the CSG syntax files together with the Java parser. The creation of the CSGPVisitor accessor and CSGPParser parser, combined with the Protocol object and preset parsing rules, parses byte stream data. This includes: creating a CSGPVisitor accessor interface to define the processing logic for different column types and the overall protocol during parsing; creating a CSGPParser parser to recursively access the column definitions of the Protocol object, buffering the column data during the first access, retrieving data from the cached object when encountering a length reference column, and returning the column information and parsed data to the user through the CSGPVisitor accessor interface; and integrating the functionality of the CsgVisitor class into the CSGPParser parser to implement the user's access logic for the parsed data, and parsing the byte stream data by combining the Protocol object and preset parsing rules.

2. The data parsing method based on a syntax analyzer generator according to claim 1, characterized in that, The process of creating a CSG syntax file based on the syntax specification of the syntax analyzer, defining the context-dependent syntax rules of the protocol, includes: Define CSG syntax metadata information, including the name, version, comments, and protocol syntax definition of the syntax; Provides the content definition of the protocol, including the content column definition and the syntax definition that distinguishes between ordinary columns and combined columns; Define attributes for regular columns and composite columns separately. Regular column attributes include column ID, name, data type, data length, rendering type, and remarks; composite column attributes include column ID, name, rendering type, selected field, and data content. It provides basic lexical and grammatical definitions.

3. The data parsing method based on a syntax analyzer generator according to claim 2, characterized in that, The aforementioned Java parser, based on the generated code, implements a custom CsgVisitor class to parse CSG syntax files and construct Protocol objects, including: Use a Java parser to parse the abstract syntax tree of a CSG syntax file; Define a class CsgVisitor that inherits from the CsgBaseVisitor class generated by the parser generator; The CsgVisitor class is used to recursively call the visit method, traversing the abstract syntax tree of the CSG syntax file from child nodes to the root node, and constructing a Protocol object; where child nodes return the base object, and the root node aggregates the results of child nodes.

4. A data parsing device based on a syntax analyzer generator, characterized in that, include: The acquisition module is used to acquire the byte stream data to be parsed in the task scenario; The definition module is used to create CSG syntax files based on the syntax specifications of the syntax analyzer generator, and to define the context-dependent syntax rules of the protocol. The generation module is used to process CSG syntax files using a syntax analysis generator, generate abstract syntax tree information related to the CSG syntax files, and generate a Java parser based on the abstract syntax tree information; The parsing module is used to implement a custom CsgVisitor class based on the generated Java parser, to parse CSG syntax files and construct Protocol objects; The module is used to create CSGPVisitor and CSGPParser accessors, which, in conjunction with Protocol objects and preset parsing rules, parse byte stream data. The process of using a syntax analyzer to process CSG syntax files, generate abstract syntax tree information related to the CSG syntax files, and generate a Java parser based on the abstract syntax tree information includes: taking a CSG syntax file that meets the requirements of the syntax analyzer as input, running the syntax analyzer to perform lexical analysis, identifying the basic elements of the language and converting them into a symbol stream; the syntax analyzer to perform semantic analysis, combining and verifying the basic elements in the symbol stream according to context-dependent grammar rules, generating abstract syntax tree information related to the CSG syntax files, and generating a Java parser based on the abstract syntax tree information; and calling CLI commands in the syntax analyzer to generate basic auxiliary data and auxiliary classes to process the CSG syntax files together with the Java parser. The creation of the CSGPVisitor accessor and CSGPParser parser, combined with the Protocol object and preset parsing rules, parses byte stream data. This includes: creating a CSGPVisitor accessor interface to define the processing logic for different column types and the overall protocol during parsing; creating a CSGPParser parser to recursively access the column definitions of the Protocol object, buffering the column data during the first access, retrieving data from the cached object when encountering a length reference column, and returning the column information and parsed data to the user through the CSGPVisitor accessor interface; and integrating the functionality of the CsgVisitor class into the CSGPParser parser to implement the user's access logic for the parsed data, and parsing the byte stream data by combining the Protocol object and preset parsing rules.

5. A data parsing server based on a syntax analyzer generator, characterized in that, Including memory and processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions, which, when executed by a computer, enable the implementation of the method as described in any one of claims 1-3.

Images