Data analysis method and device, electronic equipment and readable medium

Abstract

The present disclosure provides a data analysis method and device, electronic equipment and readable medium; it relates to network communication technical field.The data analysis method comprises the following steps: determining a message structure according to a protobuf file; when a front-end request for the protobuf file is received, obtaining to-be-analyzed field information in the front-end request, and determining a field path corresponding to the to-be-analyzed field information according to the message structure; and analyzing a target field in the protobuf file through the field path, so as to perform data processing on the target field according to the front-end request.The present disclosure can process the protobuf file without stopping and restarting, thereby improving stability.

Description

Technical Field

[0001] This disclosure relates to the field of network communication technology, and more specifically, to a data parsing method, a data parsing device, an electronic device, and a computer-readable medium. Background Technology

[0002] Protocol Buffers (protobuf) is a language- and platform-independent data exchange format. Compared to traditional data exchange formats, it offers high parsing speed and smaller file size due to its binary data transmission basis, making it suitable for various scenarios such as network transmission, configuration files, and data storage.

[0003] When communicating between the front-end and back-end, a protocol file is defined using protobuf. Each side then uses protobuf's tools to compile the protocol file into the corresponding format, ensuring consistency. If the protocol content changes, the modified protocol file needs to be recompiled, with both the front-end and back-end generating the latest compiled file. However, each time a field needs modification, the entire protobuf message needs to be requested from the back-end. After updating the required fields in the protobuf message, the entire message is repackaged into a new binary data file and returned to the back-end. Therefore, the back-end service needs to be restarted every time a field is added or modified, resulting in significant resource waste.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a data parsing method, data parsing device, electronic device, and computer-readable medium, thereby at least to some extent overcoming the problem of resource waste caused by front-end and back-end data communication due to the limitations and defects of related technologies.

[0006] According to a first aspect of this disclosure, a data parsing method is provided, comprising:

[0007] Determine the message structure based on the protobuf file;

[0008] When a frontend request for the protobuf file is received, the field information to be parsed in the frontend request is obtained, and the field path corresponding to the field information to be parsed is determined according to the message structure.

[0009] The target field in the protobuf file is parsed using the field path to process the data in the target field according to the front-end request.

[0010] In one exemplary embodiment of this disclosure, determining the message structure based on the protobuf file includes:

[0011] Obtain each object from the protobuf file, and generate a tree structure by using each object as a node through a dynamic creation method.

[0012] In one exemplary embodiment of this disclosure, the field information to be parsed includes the name of the field to be parsed; determining the field path corresponding to the field information to be parsed based on the message structure includes:

[0013] In each node of the tree structure, find the target node that matches the name of the field to be parsed, and determine the node number of the target node;

[0014] Generate a path tree for the target node based on the node label of the target node, and use the path tree as the field path.

[0015] In one exemplary embodiment of this disclosure, parsing the target field in the protobuf file via the field path includes:

[0016] Obtain the compiled binary data of the protobuf file;

[0017] The start and end boundaries of each node in the field path are parsed in the binary data to determine the target start and end boundaries of the leaf nodes in the field path.

[0018] The binary data corresponding to the target start and end boundaries is determined as the target field.

[0019] In one exemplary embodiment of this disclosure, the front-end request includes a write data request; the data processing of the target field according to the front-end request includes:

[0020] The field structure of the field to be parsed is determined using the information of the field to be parsed;

[0021] The protobuf file is updated by replacing the target field in the binary data with the field structure described above.

[0022] In one exemplary embodiment of this disclosure, the front-end request includes a read data request; the data processing of the target field according to the front-end request includes:

[0023] The aforementioned binary data is transmitted to the front end as the target field.

[0024] In one exemplary embodiment of this disclosure, the method further includes:

[0025] A request interface for the protobuf file is generated to receive the front-end request through the request interface, wherein the request interface includes a read request and an update request for the protobuf file.

[0026] According to a second aspect of this disclosure, a data parsing apparatus is provided, which may include a message structure determination module, a field path determination module, and a target field processing module. Wherein:

[0027] The message structure determination module is used to determine the message structure based on the protobuf file; the field path determination module is used to obtain the field information to be parsed in the front-end request when a front-end request for the protobuf file is received, and determine the field path corresponding to the field information to be parsed based on the message structure; the target field processing module is used to parse the target field in the protobuf file through the field path, so as to perform data processing on the target field according to the front-end request.

[0028] In one exemplary embodiment of this disclosure, the message structure determination module may be configured to: obtain each object in the protobuf file, and generate a tree structure by using each object as a node through a dynamic creation method.

[0029] In one exemplary embodiment of this disclosure, the field information to be parsed includes the name of the field to be parsed; the field path determination module includes:

[0030] The node label determination module is used to find the target node that matches the name of the field to be parsed in each node of the tree structure and determine the node label of the target node; the path tree generation module is used to generate the path tree of the target node based on the node label of the target node and use the path tree as the field path.

[0031] In one exemplary embodiment of this disclosure, the target field processing module includes:

[0032] The data acquisition module is used to acquire the binary data of the protobuf file after compilation; the boundary division module is used to parse the start and end boundaries of each node in the field path in the binary data to determine the target start and end boundaries corresponding to the leaf nodes in the field path; the target field determination module is used to determine the part of the binary data corresponding to the target start and end boundaries as the target field.

[0033] In one exemplary embodiment of this disclosure, the front-end request includes a write data request; the target field processing module includes:

[0034] The field structure determination module is used to determine the field structure of the field to be parsed using the field information to be parsed; the field update module is used to replace the target field in the binary data with the field structure to update the protobuf file.

[0035] In one exemplary embodiment of this disclosure, the front-end request includes a read data request; the target field processing module is specifically used to transmit the partial binary data as the target field to the front-end.

[0036] In one exemplary embodiment of this disclosure, the data parsing apparatus further includes:

[0037] An interface generation module is used to generate a request interface for the protobuf file, so as to receive the front-end request through the request interface, wherein the request interface includes a read request and an update request for the protobuf file.

[0038] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method described in any of the preceding methods by executing the executable instructions.

[0039] According to a fourth aspect of this disclosure, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0040] According to a fifth aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable medium. A processor of a computer device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the computer device to perform the data parsing methods provided in the various embodiments described above.

[0041] The exemplary embodiments disclosed herein may have some or all of the following beneficial effects:

[0042] In the data parsing method provided in an example embodiment of this disclosure, when a protobuf file is requested, only the target fields that need to be updated in the protobuf file can be parsed, without parsing the entire message. On the one hand, this avoids unnecessary waste of resources and saves computing resources; on the other hand, each request does not need to obtain the complete protobuf file from the backend, but only the target fields in the protobuf file need to be processed, which can save bandwidth and improve data transmission efficiency; furthermore, when processing data in the protobuf file, the backend can provide services without restarting, which can improve stability and has a wider range of application scenarios.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0045] Figure 1 A schematic diagram of an exemplary system architecture for a data parsing and apparatus to which embodiments of the present disclosure can be applied is shown.

[0046] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0047] Figure 3 A flowchart illustrating a data parsing method according to an embodiment of the present disclosure is shown schematically.

[0048] Figure 4 A schematic diagram of a message structure according to one embodiment of the present disclosure is shown.

[0049] Figure 5 A schematic diagram of a message structure according to another embodiment of this disclosure is shown.

[0050] Figure 6 A flowchart illustrating a data parsing method according to one embodiment of the present disclosure is shown schematically.

[0051] Figure 7 A schematic diagram of node labels is shown in one embodiment of the present disclosure.

[0052] Figure 8A schematic diagram of a path tree according to one embodiment of the present disclosure is shown.

[0053] Figure 9 A flowchart illustrating a data parsing method according to one embodiment of the present disclosure is shown schematically.

[0054] Figure 10 A schematic diagram illustrating data parsing according to one embodiment of the present disclosure is shown.

[0055] Figure 11 A schematic diagram illustrating data parsing according to another embodiment of this disclosure is shown.

[0056] Figure 12 A block diagram of a data parsing apparatus according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0057] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0058] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0059] Figure 1 A schematic diagram of a system architecture for an exemplary application environment in which an embodiment of the data parsing method and data parsing apparatus of the present disclosure can be applied is shown.

[0060] like Figure 1As shown, system architecture 100 may include one or more of terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables. Terminal devices 101, 102, and 103 may be various electronic devices with displays, including but not limited to desktop computers, laptops, smartphones, and tablets. It should be understood that... Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0061] The data parsing method provided in this embodiment is generally executed by server 105, and correspondingly, the data parsing device is generally disposed in server 105. However, it is readily understood by those skilled in the art that the data parsing method provided in this embodiment can also be executed by terminal devices 101, 102, and 103, and correspondingly, the data parsing device can also be disposed in terminal devices 101, 102, and 103. This exemplary embodiment does not impose any special limitations on this.

[0062] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0063] It should be noted that, Figure 2 The computer system 200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0064] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 202 or programs loaded from storage section 208 into random access memory (RAM) 203. The RAM 203 also stores various programs and data required for system operation. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0065] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, modem, etc. The communication section 209 performs communication processing via a network such as the Internet. Drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0066] In particular, according to embodiments of this disclosure, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs the various functions defined in the methods and apparatus of this application.

[0067] The technical solutions of the embodiments of this disclosure are described in detail below:

[0068] Adding a new message structure to Protobuf requires compilation. If the backend needs to maintain consistency with the protobuf messages used by the frontend, the backend service needs to recompile and link the latest protobuf files and restart. Updating even a small nested field in the message structure requires retrieving the entire protobuf message from the backend, unpacking the message, parsing each field individually, locating the field to be updated, updating that field, and then repackaging it, which incurs significant performance overhead.

[0069] Technical terms:

[0070] A Protocol Buffer file is a source file defined according to the Protocol Buffer specification, ending with .proto. The Protocol file defines the message structure through messages, including the message name, field names, field types, and unique identifiers for each field within the message structure, forming a TLV structure. Here, T represents the unique identifier (tag) of the data; L represents the length of the data; and V represents the value.

[0071] Protobuf messages refer to message structures generated from .proto files, which can be packaged into binary form according to the protobuf specification for transmission or storage.

[0072] To address one or more of the aforementioned problems, this exemplary embodiment provides a data parsing method. This data parsing method can be applied to the server 105, or to one or more of the terminal devices 101, 102, and 103, and is not specifically limited in this exemplary embodiment. (See reference...) Figure 3 As shown, this data parsing method may include the following steps:

[0073] Step S310. Determine the message structure based on the protobuf file.

[0074] Step S320. When a front-end request for the protobuf file is received, obtain the field information to be parsed in the front-end request, and determine the field path corresponding to the field information to be parsed according to the message structure.

[0075] Step S330. Parse the target field in the protobuf file through the field path to process the data of the target field according to the front-end request.

[0076] In the data parsing method provided in this example implementation, when a protobuf file is requested, only the target fields that need to be updated in the protobuf file can be parsed, without parsing the entire message. On the one hand, this avoids unnecessary waste of resources and saves computing resources; on the other hand, each request does not need to obtain the complete protobuf file from the backend, but only the target fields in the protobuf file need to be processed, which can save traffic and improve data transmission efficiency; furthermore, when processing data in the protobuf file, the backend can provide services without restarting, which can improve stability and has a wider range of application scenarios.

[0077] The steps described above in this example implementation will now be explained in more detail.

[0078] Step S310. Determine the message structure based on the protobuf file.

[0079] In a protobuf file, messages are defined in code. A message can consist of at least one field, and each field can have a different format. In this exemplary embodiment, the latest protobuf file is uploaded to the server. When the server receives the protobuf file, it can parse it using the corresponding API to obtain the message structure defined in the file. The front-end and back-end exchange data using this message structure.

[0080] Protobuf provides corresponding APIs (Application Programming Interfaces) to parse protobuf files. For example, the latest .proto file can be packaged into a text file and uploaded to the server. The server can use the interface google::protobuf::io::compiler::Importer to parse the contents of the text file and obtain the message. A message can include multiple fields or substructures, and each substructure also consists of at least one field, thus forming a complete message structure.

[0081] For example, a protobuf file is defined as follows:

[0082]

[0083] Parsing the protobuf file yields the following: Figure 4 The message structure is shown below. This protobuf file defines a message named msg, which includes five objects: id, name, pay, location, and login. The pay and location objects have their own substructures, such as... Figure 4 As shown, substructures can be nested within substructures, such as the account object contained in pay.

[0084] In an exemplary implementation, each object in the protobuf file is obtained, and a tree structure is generated using the obtained objects as nodes through a dynamic creation method. Specifically, by Figure 4 As can be seen, the message structure exhibits a distinct tree-like structure. Therefore, we can first read the various objects contained in the protobuf file, and then dynamically create and insert each object into the tree structure. This dynamic creation method allows for timely updates to the message structure when the file changes, ensuring consistency between the front-end and back-end message structures without requiring a server restart. This improves stability and meets the needs for updating protobuf.

[0085] For example, protobuf supports reflection, so it's possible to obtain individual objects from a protobuf message, generate a tree structure using the DynamicMessageFactory method, and then reflectively obtain a tree structure consistent with the message structure in memory. Figure 5 As shown, objects are described using Descriptors, and each object can be a field. Substructures also have field members. The final field, i.e., the field that does not contain substructures, is described using FieldDescriptors. Each field has a type: Full_name represents the field name, and type represents the field type. The field type determines the size, thus determining the size of the protobuf message.

[0086] It should be noted that the Descriptor class and FieldDescirptor class are used to describe the data in this exemplary embodiment. In addition, other classes can also be used to describe the data, such as DescriptorPool, FileDescirptor, etc. This embodiment is not limited to these.

[0087] Step S320. When a front-end request for the protobuf file is received, obtain the field information to be parsed in the front-end request, and determine the field path of the field information to be parsed according to the message structure.

[0088] Here, a front-end request can refer to a request sent by the client to the server. Specifically, it can include the request type, the field names of the request, and information about the client and the server, such as the client's ID, IP address, server address, etc. This implementation does not impose any special limitations on this.

[0089] If a client needs to read a protobuf message or a specific field within it, the front-end (client) can send a read request to the server. When the server receives this request, it can parse it to retrieve the field information to be parsed. This field information may include at least one field, specifically its name or content. Furthermore, depending on the specific requirements, it may also include other information, such as the field type; this implementation does not impose any special limitations on this. The field path corresponding to the field information can be understood as the query path of the field within the message structure. Based on this path, the position of the field within the message structure can be determined, allowing the corresponding field to be read.

[0090] The method for determining the field path of the field information to be parsed based on the message structure may include steps S610 and S620, such as... Figure 6 As shown.

[0091] Step S610. Find the target node that matches the name of the field to be parsed in each node of the tree structure, and determine the node number of the target node.

[0092] Step S620. Generate a path tree for the target node based on the node label of the target node, and use the path tree as the field path.

[0093] In step S610, within the tree structure constructed based on the message, each node is matched starting from the root node. Each node in this tree structure represents a field in the protobuf message. The field names contained within the node are matched against the name of the field to be parsed, thereby identifying the target node that matches the name of the field to be parsed. After finding the target node, its node label is determined. The node label can include the identifier of the path from the root node to the child node, or the node's identifier. If the field to be parsed is a nested field, then the name of the field to be parsed contains multiple nodes, meaning there are multiple target nodes that match the name of the field to be parsed. Each node in this tree structure has a node label, which is a unique identifier for the node in the tree structure; different nodes have different node labels. When generating a tree structure, each node can be assigned a unique node number. For example, the node number of the first child node of the root node can be 1, the node number of the second child node of the root node can be 2, and so on. For another example, if the first child node of the root node also has substructures, then the node number of the first child node of that node can be 1.1, the node number of the second child node of that node can be 1.2, and so on. If there are further substructures, then the node numbers of each node in its substructures are also numbered in this way.

[0094] For example, if the node numbers of each node in a tree structure are as follows: Figure 7As shown, the root node contains 5 child nodes. The child nodes 1, 2, 3, 4, and 5 of the root node are matched sequentially with the names of the fields to be parsed. If no matching target node is found, the search continues down the next level to the child nodes of child nodes 2, 3, and 4. Assuming the name of the field to be parsed is the same as the field name of node T, then node T is the target node, and its node number is 3.3.2. If there are multiple fields to be parsed, each field name can be used to determine a corresponding target node. Assuming one field name matches node T and another matches node G, then the target nodes are nodes T and G, with the node number of target node T being 3.3.2 and the node number of target node G being 4.

[0095] In step S620, the structure of the subtree containing the target node can be reconstructed based on the node index of the target node, serving as the path tree for the target node. Each node in this path tree has its index below the root node, thus a root node with a value of zero can be generated for the path tree, ensuring that the structure of the path tree matches the message structure in memory. The path tree is a subtree of a tree structure and can serve as the field path for the fields to be parsed, including paths from the root node of the tree structure to each target node. For example, combining... Figure 4 and Figure 7 If the fields to be parsed are "pay.account.balance" and "location", and their corresponding target node numbers are 3.3.2 and 4 respectively, then the path tree reconstructed based on the target node numbers can be as follows: Figure 8 As shown. Specifically, a root node with a value of zero is first generated. Then, the first parent node in the node index is retrieved and searched among the child nodes of this root node. Initially, the root node has no child nodes, so the first parent node is inserted into the child nodes of the root node. The first parent node in the node index of pay.account.balance is 3, so the first child node of the root node is 3. The first parent node in the node index of location is 4, so the other child node of the root node is 4. At this point, the root node with a value of zero has two child nodes, 3 and 4. These two child nodes are used as the parent nodes of the substructure, and the first parent node mentioned above in the node index is removed. The above steps are repeated until the node index is empty, generating the path tree of the target node. This path tree is a subtree relative to the original protobuf message structure.

[0096] Step S330. Parse the target field in the protobuf file through the field path to process the data of the target field according to the front-end request.

[0097] The field path can be used to determine the part of the message structure corresponding to that field path, allowing only that part to be parsed to obtain the target field that needs to be parsed. Since the Protobuf file is stored as binary data in memory, reading this file requires first parsing the corresponding binary data into the required format before reading the data. This method includes steps S910, S920, and S930, as follows... Figure 9 As shown. Wherein:

[0098] Step S910. Obtain the compiled binary data of the protobuf file. After receiving the protobuf file written in code, the server can provide the compiler provided by protobuf to compile the file into binary data for storage. The corresponding binary data can be read from memory according to the message names defined in the protobuf file.

[0099] Step S920. Parse the start and end boundaries of each node in the field path corresponding to the binary data to determine the target start and end boundaries corresponding to the leaf nodes in the field path. The field path includes a path pointing to the target node, which can connect multiple nodes. By mapping the field path to the message structure, the fields corresponding to the field path in the message structure can be determined, thereby determining the start and end boundaries of each field based on the characteristics of protobuf. In an exemplary embodiment, the path tree structure of the field path can be mapped to the binary data to determine the boundaries corresponding to each node in the path tree in the binary data, thereby dividing the binary data into multiple fields. Specifically, the root node of the field path points to the binary data, and then it is determined whether the root node has child nodes to obtain its first-level child nodes. Then, each field in the binary data is parsed sequentially. Based on the TLV characteristics of protobuf, only the type and length of each field, i.e., tag and length, can be parsed to determine the fields corresponding to the first-level child nodes. The nodes in the field path are matched with the parsed fields to determine the fields corresponding to each node in the field path and the start and end boundaries of the fields. For irrelevant fields in binary data, only the type and length need to be parsed, without parsing the field values. This process is repeated recursively until the target start and end boundaries corresponding to the leaf nodes in the field path are found, thus completing the parsing.

[0100] Step S930. Determine the portion of binary data corresponding to the target start and end boundaries as the target field. The portion of binary data within the target start and end boundaries of the leaf nodes in the field path is the target field corresponding to the field information to be parsed. Based on the front-end request, the complete data packet of the target field can be parsed to determine the value of the target field, and the value of the target field can be returned to the front-end request. Alternatively, other data processing can be performed on the target field based on the front-end request. For example, combined with... Figure 10 The binary data corresponding to the field pay.account.balance to be parsed is 1020, and the binary data corresponding to location is 1030. Both of these binary data can be identified as the target fields.

[0101] For binary data, existing technologies require parsing the entire binary data. However, this exemplary implementation can determine the boundaries of each node based on the field path, and only parse the type and length of the node without parsing the value of each field. For leaf nodes in the field path, even if they are actually nested fields, there is no need to parse the nested fields they contain. This can greatly reduce the amount of computation, save computing resources, and improve the utilization rate of computing resources.

[0102] Figure 10 This diagram illustrates the process of parsing binary data and determining the start and end boundaries of each node. (Reference) Figure 10 As shown, for field paths 3.3.2 and 4, their root nodes are set to binary data A. The binary data is then parsed sequentially from the beginning. First, the first field is parsed, and the parsed type tag is matched against the first parent node in the field path. The parent node's name is compared with the first field's name to determine if they match. If they match, the start and end boundaries are determined based on the first field's length. If the first field is a leaf node in the field path, its start boundary includes the head of the binary data and a tail of length length from the head. This start and end boundary is the target start and end boundary. If the first field does not match the first parent node in the field path, the second field is parsed, and so on, until a field matching the first parent node is found. The head of this field is determined, and its tail is determined based on its length, thus obtaining the boundary of the first parent node. For example, in field path 3.3.2, the start and end boundaries of the first parent node 3 are a1 and b1, respectively, where the start boundary is a1 and the end boundary is b1. For the binary data from the beginning up to the part before node 3 (1010), only the type and length of each field need to be parsed, without parsing the data packet; that is, the data 1010 part is the unpacked part.

[0103] After matching the first parent node in the field path, if the first parent node is the last node in the field path (i.e., a leaf node, meaning it does not contain substructures), then the parsing is complete. If the first parent node is not a leaf node, then the data within the start and end boundaries of that first parent node is parsed recursively using the above parsing method. That is, starting from the starting position 'a' of the first parent node, the type and length of each field it contains are parsed, and each field is matched with the second parent node in the field path to determine the start and end boundaries corresponding to the second parent node. This process continues until the start and end boundaries of the last node in the field path are determined. The start and end boundaries of the last node are the target start and end boundaries. (Reference) Figure 10 For the third node in field path 3.3.2, which is a leaf node, the target start and end boundaries are a2 and b2. For field path 4, which is the last node in the field path, the corresponding target start and end boundaries are a3 and b3. Even if this node contains child nodes, there is no need to parse it again.

[0104] After parsing the target start and end boundaries of the target field, data processing can be performed on the target field according to the type of front-end request. Front-end requests can have various types, such as read data requests and write data requests. When the front-end request is a write data request, the target field can be updated using the parsed field information; when the front-end request is a read data request, the target field can be sent to the front-end.

[0105] Specifically, if the frontend request is a read data request, the aforementioned binary data can be transmitted to the frontend as the target field. Based on the target start and end boundaries of the target field, the data within those boundaries is read from the saved protobuf binary data. This data is a TLV structure, and the read data is returned to the frontend. After receiving the binary data, the frontend can parse it and convert it into a user-readable format for display.

[0106] If the frontend request is a write data request, the field structure of the field to be parsed is determined using the information of the field to be parsed. The target field in the binary data is then replaced with this field structure to update the protobuf file. Based on the information of the field to be parsed, the TLV structure of the field to be parsed can be determined, i.e., the type, length, and value of the field. The information of the field to be parsed is then packaged according to the TLV structure, and the packaged TLV data is inserted into the binary data. For example... Figure 11 As shown, the original data is replaced with the packaged data 1110 and 1120, and binary data is regenerated to complete the update of the protobuf file on the backend, so that the protobuf file on the backend is consistent with the frontend.

[0107] In an exemplary embodiment, the data parsing method provided in this disclosure further includes generating a request interface for the protobuf file to receive front-end requests through the request interface. The request interface includes read and update requests for the protobuf file. To facilitate front-end processing of the protobuf file, the algorithms in steps S310 to S330 can be encapsulated to generate the request interface. Multiple request interfaces can be generated according to different requirement types, such as interfaces for reading data and interfaces for writing data. This request interface can read and update the protobuf file through field paths. For example, if a client needs to update a specified field in `mag` of a protobuf message on the server, the request interface is called using the field path `fielda.fieldb.fieldc` of the specified field: `int FieldSet(const std::set`. <std::string>&dottedpaths,::google::protobuf::Message*msg), where "FieldSet" is the name of the request interface, and "const std::set <std::string>"&dottedpaths,::google::protobuf::Message*msg" is the input parameter for this request interface. "dottedpaths" is used to receive the field paths; in this embodiment, the value of this parameter is fielda.fieldb.fieldc. For example, when the client needs to read the value of a specific field from a protobuf message, the request interface can specifically be "int FieldGet(const std::set... <std::string>&dottedpaths,::google::protobuf::Message*msg,std::set <std::string>*failedpaths), where "FieldGet" is the name of the requested interface, and "const std::set <std::string>&dottedpaths,::google::protobuf::Message*msg,std::set <std::string>"*failedpaths" is the input parameter for this request interface, where failedpaths represents the paths of fields that do not exist in the request.

[0108] It should be noted that in this exemplary embodiment, the request interface is divided into a read data interface FieldGet and a write data interface FieldSet. However, in other embodiments of this disclosure, the request interface can also be other types of interfaces. The interface name, access method, etc. can be set according to actual needs, and these also fall within the protection scope of this disclosure.

[0109] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0110] Furthermore, this example embodiment also provides a data parsing apparatus. This data parsing apparatus can be applied to a server or terminal device. (See reference...) Figure 12 As shown, the data parsing device 1200 may include a message structure determination module 1210, a field path determination module 1220, and a target field processing module 1230. Wherein:

[0111] The message structure determination module 1210 is used to determine the message structure based on the protobuf file; the field path determination module 1220 is used to obtain the field information to be parsed in the front-end request when a front-end request for the protobuf file is received, and determine the field path corresponding to the field information to be parsed based on the message structure; the target field processing module 1230 is used to parse the target field in the protobuf file through the field path, so as to perform data processing on the target field according to the front-end request.

[0112] In one exemplary embodiment of this disclosure, the message structure determination module 1210 may be configured to: obtain each object in the protobuf file, and generate a tree structure by using each object as a node through a dynamic creation method.

[0113] In one exemplary embodiment of this disclosure, the field information to be parsed includes the name of the field to be parsed; the field path determination module 1220 includes:

[0114] The node label determination module is used to find the target node that matches the name of the field to be parsed in each node of the tree structure and determine the node label of the target node; the path tree generation module is used to generate the path tree of the target node based on the node label of the target node and use the path tree as the field path.

[0115] In one exemplary embodiment of this disclosure, the target field processing module 1230 includes:

[0116] The data acquisition module is used to acquire the binary data of the protobuf file after compilation; the boundary division module is used to parse the start and end boundaries of each node in the field path in the binary data to determine the target start and end boundaries corresponding to the leaf nodes in the field path; the target field determination module is used to determine the part of the binary data corresponding to the target start and end boundaries as the target field.

[0117] In one exemplary embodiment of this disclosure, the front-end request includes a write data request; the target field processing module 1230 includes:

[0118] The field structure determination module is used to determine the field structure of the field to be parsed using the field information to be parsed; the field update module is used to replace the target field in the binary data with the field structure to update the protobuf file.

[0119] In one exemplary embodiment of this disclosure, the front-end request includes a read data request; the target field processing module 1230 is specifically used to transmit the partial binary data as the target field to the front-end.

[0120] In one exemplary embodiment of this disclosure, the data parsing apparatus 1200 further includes:

[0121] An interface generation module is used to generate a request interface for the protobuf file, so as to receive the front-end request through the request interface, wherein the request interface includes a read request and an update request for the protobuf file.

[0122] The data parsing device provided in this embodiment does not require a shutdown and restart when the front end needs to update or read the protobuf file. It can update the target fields in the protobuf file according to the needs of the front end, which can reduce the consumption of computing resources and thus improve processing efficiency.

[0123] The specific details of each module or unit in the aforementioned data parsing device have been described in detail in the corresponding data parsing methods, so they will not be repeated here.

[0124] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0125] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0128] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0129] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.< / std::string> < / std::string> < / std::string> < / std::string> < / std::string> < / std::string>

Claims

1. A data parsing method, characterized in that, include: The message structure is determined based on the protobuf file; wherein the protobuf file includes a message, and the message includes at least one field; the protobuf file supports reflection mechanisms; When a frontend request for the protobuf file is received, the information of the fields to be parsed in the frontend request is obtained, and the field path corresponding to the information of the fields to be parsed is determined according to the message structure; wherein, the frontend request includes a read data request and a write data request, the read data request is used to read the messages in the protobuf file; the write data request is used to update the messages in the protobuf file; Obtain the compiled binary data of the protobuf file; The start and end boundaries of each node in the field path are parsed in the binary data to determine the target start and end boundaries of the leaf nodes in the field path. The binary data corresponding to the target start and end boundaries is determined as the target field, and the target field is processed according to the front-end request.

2. The method according to claim 1, characterized in that, The step of determining the message structure based on the protobuf file includes: Obtain each object from the protobuf file, and generate a tree structure by using each object as a node through a dynamic creation method.

3. The method according to claim 2, characterized in that, The field information to be parsed includes the name of the field to be parsed; determining the field path corresponding to the field information to be parsed based on the message structure includes: In each node of the tree structure, find the target node that matches the name of the field to be parsed, and determine the node number of the target node; Generate a path tree for the target node based on the node label of the target node, and use the path tree as the field path.

4. The method according to claim 1, characterized in that, The front-end request is a write data request; the data processing of the target field according to the front-end request includes: The field structure of the field to be parsed is determined using the information of the field to be parsed; The protobuf file is updated by replacing the target field in the binary data with the field structure described above.

5. The method according to claim 1, characterized in that, The front-end request is a read data request; the data processing of the target field based on the front-end request includes: The aforementioned binary data is transmitted to the front end as the target field.

6. The method according to claim 1, characterized in that, The method further includes: A request interface for the protobuf file is generated to receive the front-end request through the request interface, wherein the request interface includes a read request and an update request for the protobuf file.

7. A data parsing device, characterized in that, include: A message structure determination module is used to determine the message structure based on a protobuf file; wherein the protobuf file includes a message, and the message includes at least one field; the protobuf file supports reflection mechanisms; The field path determination module is used to, upon receiving a front-end request for the protobuf file, obtain the field information to be parsed in the front-end request, and determine the field path corresponding to the field information to be parsed based on the message structure; wherein, the front-end request includes a read data request and a write data request, the read data request is used to read messages in the protobuf file; the write data request is used to update messages in the protobuf file; The target field processing module is used to obtain the binary data of the protobuf file after compilation; parse the start and end boundaries of each node in the field path in the binary data to determine the target start and end boundaries corresponding to the leaf nodes in the field path; determine the part of the binary data corresponding to the target start and end boundaries as the target field, so as to process the target field according to the front-end request.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the data parsing method according to any one of claims 1-6 by executing the executable instructions.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data parsing method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product or computer program includes computer instructions stored in a computer-readable medium; The processor of the computer device reads the computer instructions from a computer-readable medium, and executes the computer instructions to cause the computer device to perform the data parsing method according to any one of claims 1-6.

Images