Data conversion method and device of system data application layer, equipment and storage medium

Abstract

The invention relates to a data conversion method and device of a system data application layer, computer equipment and a storage medium. The method comprises the steps that a data model is determined; dynamically loading a file analysis component based on the file format type of the to-be-converted data file; analyzing the to-be-converted data file through a file analysis component to obtain metadata of an initial data object of the data model, configuring the metadata of the initial data object into the data model, and configuring each data conversion instruction of the data model based on the relationship of each data object to obtain a target model file; analyzing each data object, a data conversion instruction corresponding to each data object and metadata of the initial data object from the target model file, and constructing an abstract syntax tree; and traversing the abstract syntax tree to obtain a plurality of target nodes and edges connected with the target nodes, and executing data conversion processing based on a traversing result. According to the method, logic codes do not need to be written for different business requirements and different data formats, and the data conversion efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data conversion method, device, computer equipment and storage medium at a system data application layer. Background Art

[0002] Currently, when the system data application layer is processing data conversion, the code is written based on the specific data conversion logic, and the data is converted by running the logic code. For different data conversion logics, the system needs to develop corresponding logic codes.

[0003] However, in response to a large number of different business needs, the data that needs to be converted has a variety of data formats. Based on specific business needs and different data formats, multiple sets of logic codes need to be developed. This not only consumes more time and manpower, but also has low data conversion efficiency at the system data application layer. Summary of the invention

[0004] Based on this, it is necessary to provide a data conversion method, device, computer equipment and storage medium for the system data application layer to address the above technical problems, which can abstract the data conversion logic from the specific business and perform automatic conversion through data models and abstract syntax trees. This eliminates the need to write logic codes for different business needs and different data formats, thereby improving the efficiency of data conversion.

[0005] A data conversion method for a system data application layer comprises: determining a data model, the data model is used to describe the data structure of each data object and the relationship between each data object; identifying the file format type of a data file to be converted, and dynamically loading a file parsing component based on the file format type; parsing the data file to be converted by the file parsing component, obtaining metadata of an initial data object of the data model from the parsing result, configuring the metadata of the initial data object into the data model, and configuring each data conversion instruction of the data model based on the relationship between each data object, and obtaining a target model file after configuration; parsing each data object and the data conversion instruction corresponding to each data object and the metadata of the initial data object from the target model file by a model parser, constructing an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node; traversing the abstract syntax tree to obtain a plurality of target nodes and edges connecting each target node, and performing data conversion processing based on the plurality of target nodes and the edges connecting each target node and the metadata of the initial data object.

[0006] In one of the embodiments, the target file parsing component is dynamically loaded based on the file format type, including: if the file format type is JSON type, the loaded target file parsing component is a JACKSON component; if the file format type is YAML type, the loaded target file parsing component is a SnakeYAML component; if the file format type is XML type, the loaded target file parsing component is a JAXP component.

[0007] In one embodiment, after the step of constructing an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node, it also includes: obtaining the data structure of each node of the abstract syntax tree, and determining the node type of each node based on the data structure of each node; constructing a hash table, and creating each index of the hash table based on each node type; traversing the abstract syntax tree to obtain multiple target nodes and edges connecting each target node, including: traversing the abstract syntax tree based on each index of the hash table to obtain multiple target nodes and edges connecting each target node.

[0008] In one of the embodiments, data conversion processing is performed based on multiple target nodes and edges connecting each target node, and metadata of an initial data object, including: determining a first target node and multiple second target nodes among the multiple target nodes, the first target node being a node of a first object to be converted, and the multiple second target nodes including a node of the initial data object and each conversion node before the first target node; performing data conversion processing based on edges connecting each target node and metadata of the initial data object to obtain conversion data of the object to be converted of the first target node.

[0009] In one of the embodiments, data conversion processing is performed based on multiple target nodes and edges connecting each target node, and metadata of an initial data object, and also includes: determining a third target node and multiple fourth target nodes among the multiple target nodes, the third target node being a node of a second object to be converted, and the multiple fourth target nodes including a node of the initial data object and each conversion node before the third target node; starting a first thread and a second thread for parallel processing; controlling the first thread to run data conversion processing of the first target node; and controlling the second thread to run data conversion processing of the third target node.

[0010] In one of the embodiments, a data conversion method of a system data application layer further includes: caching data of a first object to be converted obtained after data conversion processing of a first target node; and caching data of a second object to be converted obtained after data conversion processing of a third target node.

[0011] In one of the embodiments, after obtaining the target model file, it also includes: determining the version of the target model file; recording the version of the target model file in a version information table of a database, wherein the version information table records the versions of one or more model files, and the differences between the versions of the model files include the updating, addition or deletion of some data; reading the corresponding template file based on any version in the version information record table, parsing each data object and the data conversion instructions corresponding to each data object, and the metadata of the initial data object from the corresponding template file through a model parser to perform data conversion processing.

[0012] A data conversion device for a system data application layer comprises: a determination module for determining a data model, wherein the data model is used to describe the data structure of each data object and the relationship between each data object; a loading module for identifying the file format type of a data file to be converted, and dynamically loading a file parsing component based on the file format type; a parsing and configuration module for parsing the data file to be converted through the file parsing component, obtaining metadata of an initial data object of the data model from the parsing result, configuring the metadata of the initial data object into the data model, and configuring each data conversion instruction of the data model based on the relationship between each data object, and obtaining a target model file after configuration; a construction module for parsing each data object and the data conversion instruction corresponding to each data object and the metadata of the initial data object from the target model file through a model parser, and constructing an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node; a data conversion module for traversing the abstract syntax tree to obtain a plurality of target nodes and edges connecting each target node, and performing data conversion processing based on the plurality of target nodes and the edges connecting each target node and the metadata of the initial data object.

[0013] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned embodiments when executing the computer program.

[0014] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned embodiments of the method are implemented.

[0015] The above-mentioned data conversion method, device, computer equipment and storage medium of the data application layer of a system determine a data model, which is used to describe the data structure of each data object and the relationship between each data object; identify the file format type of the data file to be converted, and dynamically load the file parsing component based on the file format type; parse the data file to be converted by the file parsing component, obtain the metadata of the initial data object of the data model from the parsing result, configure the metadata of the initial data object into the data model, and configure each data conversion instruction of the data model based on the relationship between each data object, and obtain the target model file after configuration; parse each data object and the data conversion instruction corresponding to each data object and the metadata of the initial data object from the target model file by the model parser, and construct an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node; traverse the abstract syntax tree to obtain multiple target nodes and the edges connecting each target node, and perform data conversion processing based on the multiple target nodes and the edges connecting each target node and the metadata of the initial data object.

[0016] Therefore, the metadata to be converted can be parsed from the data file to be converted, and then the data conversion instructions corresponding to the data conversion logic can be configured through the data model. The data conversion logic can be abstracted from the specific implementation without developing code. In addition, an abstract syntax tree is constructed based on this, and data conversion is performed by traversing the abstract syntax tree, which can improve the efficiency of data conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 An application environment diagram of a data conversion method for a system data application layer in an embodiment;

[0018] Figure 2 A schematic diagram of a flow chart of a data conversion method of a system data application layer in an embodiment;

[0019] Figure 3 A schematic diagram of a partial code structure of a data model in one embodiment;

[0020] Figure 4 A schematic diagram of another part of the code structure of a data model in one embodiment;

[0021] Figure 5 A structural block diagram of a data conversion device of a system data application layer in an embodiment;

[0022] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as including but not limited to.

[0025] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0026] The data conversion method of the system data application layer provided in this application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the data conversion system 101 is used to execute a data conversion method of the system data application layer of the present application. Specifically, the upstream system 200 transmits the data file to be converted to the data conversion system 101. The data conversion system 101 determines the data model, which is used to describe the data structure of each data object and the relationship between each data object. Identify the file format type of the data file to be converted, and dynamically load the file parsing component based on the file format type; parse the data file to be converted by the file parsing component, obtain the metadata of the initial data object of the data model from the parsing result, configure the metadata of the initial data object into the data model, and configure each data conversion instruction of the data model based on the relationship between each data object, and obtain the target model file after configuration. Among them, the data conversion instruction can be manually configured in the terminal 102. Furthermore, the data conversion system 101 parses each data object and the data conversion instructions corresponding to each data object and the metadata of the initial data object from the target model file through a model parser, and constructs an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node; traverses the abstract syntax tree to obtain multiple target nodes and the edges connecting each target node, and performs data conversion processing based on the multiple target nodes and the edges connecting each target node and the metadata of the initial data object.

[0027] In one embodiment, the present application provides a data conversion method for a system data application layer, which is applied to Figure 1 In the data conversion system 101 shown. Figure 2As shown, a data conversion method of a system data application layer includes the following steps:

[0028] S202, determining a data model, where the data model is used to describe the data structure of each data object and the relationship between each data object.

[0029] In this embodiment, a data model of metadata is defined. The structure of each data object and the relationship between each data object are defined in the data model. The structure of each data object and the relationship between each data object are represented by metadata. Specifically, the data model may include metadata of the data source, target format and conversion rules. The data source refers to the upstream system that sends the data file to be converted, which is reflected as the initial data object in the data model. The target format can also be reflected by the data object, which refers to the format of data conversion. The conversion rule is to reflect the relationship between each data object in the data model.

[0030] Specifically, the data model can be a schema model. When defining a schema model, try to simplify the structure and avoid unnecessary nesting and complex type definitions. Using clear and intuitive type names and structures will help the subsequent model parser to perform parsing more efficiently.

[0031] S204, identifying the file format type of the data file to be converted, and dynamically loading the file parsing component based on the file format type.

[0032] In one example, the above-mentioned dynamic loading of the target file parsing component based on the file format type includes: if the file format type is JSON type, the loaded target file parsing component is a JACKSON component; if the file format type is YAML type, the loaded target file parsing component is a SnakeYAML component; if the file format type is XML type, the loaded target file parsing component is a JAXP component.

[0033] Specifically, the metadata in the data model is described in JSON format, YAML format or XML format. For data files to be converted in different file formats, the corresponding components are dynamically loaded through the SPI mechanism to parse the metadata information. For example, the JSON format uses the Jackson component, the YAML format uses the SnakeYAML component, and the XML format uses the JAXP component.

[0034] After describing the metadata in the data model using JSON, YAML or XML format, it is defined as a unified schema model. That is, the data model is a schema model. Figure 3 As shown, the schema model defines the data structure of each data object and the relationship between each data object through metadata.

[0035] S206, parse the data file to be converted through the file parsing component, obtain the metadata of the initial data object of the data model from the parsing result, configure the metadata of the initial data object into the data model, and configure each data conversion instruction of the data model based on the relationship between each data object, and obtain the target model file after configuration.

[0036] In this embodiment, the data model includes multiple data objects and the relationship between each data object. The initial data object refers to the first data object among multiple data objects with relationships. Through the metadata of the initial data object, the metadata of other data objects can be determined through the relationship between each data object. Among them, the relationship between each data object is represented by each configured data conversion instruction.

[0037] S208, parse each data object and the data conversion instructions corresponding to each data object and the metadata of the initial data object from the target model file through the model parser, and construct an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node.

[0038] In this embodiment, the model parser obtains modeling information such as each data object and the data conversion instructions corresponding to each data object, metadata of the initial data object, etc. according to the data model definition, and constructs a preliminary abstract syntax tree. The parser will convert the parsed metadata into nodes of the data model and build a tree structure, create indexes and lookup tables, so as to efficiently search and operate the abstract syntax tree. Specifically, the data model is a schema model. Through the schema model, information such as the definition of data objects, the definition of operation types, and data conversion instructions can be obtained. This information is recursively used to create nodes, namely object nodes and operation types; edges, namely data conversion instructions, represent the direct conversion relationship between two data objects, thereby structuring the relationship between each data object. In the abstract syntax tree, each node represents a part of the metadata, and the edge represents the conversion relationship between nodes.

[0039] The abstract syntax tree consists of a series of nodes, each of which represents a different part of the metadata, such as type definition, field definition, etc. The following is the structure of an abstract syntax tree:

[0040] Document: The root node of the entire Schema.

[0041] Def initi on: Contains all type definitions and other declarations.

[0042] TypeDef initi on: Indicates the metadata type of the definition, such as object type, scalar type, enumeration type, etc.

[0043] objectTypeDef init i on (object type definition): represents the definition of object type, including field definition.

[0044] Field Definition: Indicates the field definition in an object type, including the field name and field type.

[0045] ScalarTypeDef init i on (scalar type definition): represents the definition of a scalar type.

[0046] EnumTypeDef init i on (enumeration type definition): represents the definition of enumeration type, including enumeration value definition.

[0047] EnumValueDefinition: A header that represents an enumeration value in an enumeration type.

[0048] InputObjectTypeDef init i on (input object type definition): represents the definition of input object type, including input field definition.

[0049] InputValueDefinition: Represents the definition of an input field in the input object type.

[0050] I nterfaceTypeDef in it i on (interface type definition): represents the definition of the interface type, including the interface field definition.

[0051] FieldDefinition: represents the field definition in the interface type.

[0052] UnionTypeDef initiation (union type definition): represents the definition of a union type, including the definitions of union type members.

[0053] DirectiveDefiniti on (instruction definition): indicates the definition of the instruction (conversion rule)

[0054] SchemaDef in it i on (Schema definition): represents the definition of Schema, including query type and change type.

[0055] OperateTypeDef init i on (operation type definition): indicates the definition of query type and change type.

[0056] NamedType: Indicates the name of the query type or change type.

[0057] D i rect i ve (instruction): indicates the use of conversion rules.

[0058] Name: The name of the command.

[0059] Argument: The parameter of the command.

[0060] S210, traversing the abstract syntax tree to obtain a plurality of target nodes and edges connecting the target nodes, and performing data conversion processing based on the plurality of target nodes, the edges connecting the target nodes, and metadata of the initial data object.

[0061] Specifically, each node of the abstract syntax tree is traversed to extract the required information. The required information includes multiple target nodes and the edges connected to each target node. According to the extracted required information, query and processing logic are generated, and the corresponding logic is executed according to the edges connected to each target node.

[0062] For example, combined with Figure 3 and Figure 4 In the abstract syntax tree, a node is constructed as TypeDef initi on (data type definition), and two nodes are connected by an "edge", namely Directive (execution). The edge corresponds to a data conversion instruction, which records how the parent node of the type definition is processed and executed to generate the child node of the type definition.

[0063] In one example, after the above step of constructing an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node, it also includes: obtaining the data structure of each node of the abstract syntax tree, and determining the node type of each node based on the data structure of each node; constructing a hash table, and creating each index of the hash table based on each node type; wherein the above traversing the abstract syntax tree to obtain multiple target nodes and edges connecting each target node includes: traversing the abstract syntax tree based on each index of the hash table to obtain multiple target nodes and edges connecting each target node.

[0064] When parsing and traversing the abstract syntax tree, you can use indexes and lookup tables to speed up node lookups. For example, create a hash table for node type definitions to quickly find nodes of a specific type. Specifically, by traversing the abstract syntax tree, create a hash table index for each node type, and by looking up the key in the hash table, you can quickly find all nodes of the corresponding type.

[0065] The above-mentioned data conversion method of the system data application layer determines a data model, which is used to describe the data structure of each data object and the relationship between each data object; identifies the file format type of the data file to be converted, and dynamically loads the file parsing component based on the file format type; parses the data file to be converted by the file parsing component, obtains the metadata of the initial data object of the data model from the parsing result, configures the metadata of the initial data object into the data model, and configures each data conversion instruction of the data model based on the relationship between each data object, and obtains the target model file after configuration; parses each data object and the data conversion instruction corresponding to each data object and the metadata of the initial data object from the target model file by the model parser, and constructs an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node; traverses the abstract syntax tree to obtain multiple target nodes and edges connecting each target node, and performs data conversion processing based on the multiple target nodes and the edges connecting each target node and the metadata of the initial data object. Therefore, the metadata to be converted can be parsed from the data file to be converted, and then the data conversion instructions corresponding to the data conversion logic can be configured through the data model. The data conversion logic can be abstracted from the specific implementation without developing code. In addition, an abstract syntax tree is constructed based on this, and data conversion is performed by traversing the abstract syntax tree, which can improve the efficiency of data conversion.

[0066] In one embodiment, the above-mentioned data conversion processing is performed based on multiple target nodes and the edges connecting each target node, and the metadata of the initial data object, including: determining a first target node and multiple second target nodes among the multiple target nodes, the first target node is a node of the first object to be converted, and the multiple second target nodes include the node of the initial data object and each conversion node before the first target node; performing data conversion processing based on the edges connecting each target node and the metadata of the initial data object to obtain conversion data of the object to be converted of the first target node.

[0067] Among them, data conversion processing is performed based on multiple target nodes and the edges connecting each target node and the metadata of the initial data object, and also includes: determining a third target node and multiple fourth target nodes among the multiple target nodes, the third target node is a node of the second object to be converted, and the multiple fourth target nodes include a node of the initial data object and each conversion node before the third target node; starting a first thread and a second thread for parallel processing; controlling the first thread to run data conversion processing of the first target node; controlling the second thread to run data conversion processing of the third target node.

[0068] Specifically, when processing the model file of the Schema model, parallel processing technology can be used to improve data processing efficiency. For example, multithreading or WebWorkers can be used to process the data conversion processing of different nodes of the abstract syntax tree. As above, when processing the data conversion of the first target node and the third target node, the first thread and the second thread of parallel processing can be used for processing.

[0069] In one example, after the step of controlling the first thread to execute the data conversion process of the first target node, the method further includes: caching the data of the first object to be converted obtained after the data conversion process of the first target node. After the step of controlling the second thread to execute the data conversion process of the third target node, the method further includes: caching the data of the second object to be converted obtained after the data conversion process of the third target node.

[0070] Specifically, the result data of the first target node and the result data of the third target node obtained after the conversion process are cached. When traversing and processing the abstract syntax tree, in order to avoid repeated calculations, the processed result data is cached to improve the efficiency of subsequent operations.

[0071] In one embodiment, after obtaining the target model file, it also includes: determining the version of the target model file; recording the version of the target model file in a version information table of a database, wherein the version information table records the versions of one or more model files, and the differences between the versions of the model files include the updating, addition or deletion of some data; reading the corresponding template file based on any version in the version information record table, parsing each data object and the data conversion instructions corresponding to each data object, and the metadata of the initial data object from the corresponding template file through a model parser to perform data conversion processing.

[0072] Specifically, Git is used to manage the model files of the Schema model. A version information table is created in the database to record the information of each version. The differences between the model files of each version include the update, addition or deletion of some data. Therefore, by selecting different versions of model files that specify different metadata to build an abstract syntax tree, it is possible to flexibly respond to changes in the data protocols required by the upstream and downstream of the data conversion layer. In addition, by creating a version information table to manage version information, it is possible to persist the metadata definition information in the historical data model and to select different versions of metadata definitions to build different abstract syntax trees for data conversion.

[0073] The data conversion method of a system data application layer of the present application can achieve the following technical effects:

[0074] Scalability: The data model defined based on metadata is applied to the data conversion layer, which can easily expand new data sources and conversion rules, so that the system can adapt to changing business needs and data formats.

[0075] Flexibility: The data model and conversion rules defined by metadata enable the system to dynamically convert data as needed without modifying the source code, which is highly flexible.

[0076] Error detection and fault tolerance: The construction and parsing process of the abstract syntax tree can detect and handle errors and anomalies in data conversion, improve the fault tolerance and stability of the system, and reduce system failures and anomalies.

[0077] Integration with existing systems: The data conversion layer defined based on metadata can be seamlessly integrated with existing systems and technology stacks without changing the existing system architecture and workflow, reducing the cost and risk of system integration.

[0078] Standardization and normalization: Data models and transformation rules defined using metadata can improve the standardization and normalization of data transformation, reduce redundancy and duplication in the system, and improve the consistency and accuracy of data.

[0079] Compatibility: The query language can well support version control and backward compatibility, making modifications to data structures easier and safer.

[0080] Performance optimization: The structure and fields of the required data can be precisely specified to avoid over- or under-acquisition and improve the efficiency of data transmission.

[0081] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0082] In one embodiment, the present application also provides a data conversion device for a system data application layer. Figure 5 As shown, a data conversion device of a system data application layer includes a determination module 502 , a loading module 504 , a parsing and configuration module 506 , a construction module 508 and a data conversion module 510 . A determination module 502 is used to determine a data model, which is used to describe the data structure of each data object and the relationship between each data object; a loading module 504 is used to identify the file format type of the data file to be converted, and dynamically load the file parsing component based on the file format type; a parsing and configuration module 506 is used to parse the data file to be converted through the file parsing component, obtain the metadata of the initial data object of the data model from the parsing result, configure the metadata of the initial data object into the data model, and configure each data conversion instruction of the data model based on the relationship between each data object, and obtain the target model file after configuration; a construction module 508 is used to parse each data object and the data conversion instruction corresponding to each data object and the metadata of the initial data object from the target model file through a model parser, and construct an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node; a data conversion module 510 is used to traverse the abstract syntax tree to obtain multiple target nodes and edges connecting each target node, and perform data conversion processing based on the multiple target nodes and the edges connecting each target node and the metadata of the initial data object.

[0083] In one of the embodiments, the target file parsing component is dynamically loaded based on the file format type, including: if the file format type is JSON type, the loaded target file parsing component is a JACKSON component; if the file format type is YAML type, the loaded target file parsing component is a SnakeYAML component; if the file format type is XML type, the loaded target file parsing component is a JAXP component.

[0084] In one embodiment, after the step of constructing an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node, it also includes: obtaining the data structure of each node of the abstract syntax tree, and determining the node type of each node based on the data structure of each node; constructing a hash table, and creating each index of the hash table based on each node type; traversing the abstract syntax tree to obtain multiple target nodes and edges connecting each target node, including: traversing the abstract syntax tree based on each index of the hash table to obtain multiple target nodes and edges connecting each target node.

[0085] In one of the embodiments, data conversion processing is performed based on multiple target nodes and edges connecting each target node, and metadata of an initial data object, including: determining a first target node and multiple second target nodes among the multiple target nodes, the first target node being a node of a first object to be converted, and the multiple second target nodes including a node of the initial data object and each conversion node before the first target node; performing data conversion processing based on edges connecting each target node and metadata of the initial data object to obtain conversion data of the object to be converted of the first target node.

[0086] In one of the embodiments, data conversion processing is performed based on multiple target nodes and edges connecting each target node, and metadata of an initial data object, and also includes: determining a third target node and multiple fourth target nodes among the multiple target nodes, the third target node being a node of a second object to be converted, and the multiple fourth target nodes including a node of the initial data object and each conversion node before the third target node; starting a first thread and a second thread for parallel processing; controlling the first thread to run data conversion processing of the first target node; and controlling the second thread to run data conversion processing of the third target node.

[0087] In one of the embodiments, a data conversion device of a system data application layer also includes a cache module for caching data of a first object to be converted obtained after data conversion processing of a first target node; and caching data of a second object to be converted obtained after data conversion processing of a third target node.

[0088] In one of the embodiments, after obtaining the target model file, it also includes: determining the version of the target model file; recording the version of the target model file in a version information table of a database, wherein the version information table records the versions of one or more model files, and the differences between the versions of the model files include the updating, addition or deletion of some data; reading the corresponding template file based on any version in the version information record table, parsing each data object and the data conversion instructions corresponding to each data object, and the metadata of the initial data object from the corresponding template file through a model parser to perform data conversion processing.

[0089] The specific definition of a data conversion device for a system data application layer can be found in the definition of a data conversion method for a system data application layer in the above text, which will not be repeated here. Each module in the above-mentioned data conversion device for a system data application layer can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0090] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external device through a network connection. When the computer program is executed by the processor, a distributed task scheduling method is implemented.

[0091] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0092] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: determining a data model, the data model being used to describe the data structure of each data object and the relationship between each data object; identifying the file format type of a data file to be converted, and dynamically loading a file parsing component based on the file format type; parsing the data file to be converted by the file parsing component, obtaining metadata of an initial data object of the data model from the parsing result, configuring the metadata of the initial data object into the data model, and configuring each data conversion instruction of the data model based on the relationship between each data object, and obtaining a target model file after configuration; parsing each data object and the data conversion instruction corresponding to each data object and the metadata of the initial data object from the target model file by a model parser, constructing an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node; traversing the abstract syntax tree to obtain a plurality of target nodes and edges connecting each target node, and performing data conversion processing based on the plurality of target nodes and the edges connecting each target node and the metadata of the initial data object.

[0093] In one embodiment, when the processor executes a computer program to implement the step of dynamically loading a target file parsing component based on a file format type, the following steps are implemented: if the file format type is a JSON type, the loaded target file parsing component is a JACKSON component; if the file format type is a YAML type, the loaded target file parsing component is a SnakeYAML component; if the file format type is an XML type, the loaded target file parsing component is a JAXP component.

[0094] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining the data structure of each node of the abstract syntax tree, and determining the node type of each node based on the data structure of each node; constructing a hash table, and creating each index of the hash table based on each node type; when the processor executes the computer program to implement the step of traversing the abstract syntax tree to obtain multiple target nodes and edges connected to each target node, the following steps are implemented: traversing the abstract syntax tree based on each index of the hash table to obtain multiple target nodes and edges connected to each target node.

[0095] In one embodiment, when a processor executes a computer program to implement the step of performing data conversion processing based on multiple target nodes and the edges connecting each target node and the metadata of the initial data object, the following steps are implemented: determining a first target node and multiple second target nodes among the multiple target nodes, the first target node being a node of the first object to be converted, and the multiple second target nodes including a node of the initial data object and each conversion node before the first target node; performing data conversion processing based on the edges connecting each target node and the metadata of the initial data object to obtain conversion data of the object to be converted of the first target node.

[0096] In one embodiment, when a processor executes a computer program to implement the step of performing data conversion processing based on multiple target nodes and the edges connecting each target node and the metadata of the initial data object, it also implements the following steps: determining a third target node and multiple fourth target nodes among the multiple target nodes, the third target node being a node of the second object to be converted, and the multiple fourth target nodes including a node of the initial data object and each conversion node before the third target node; starting a first thread and a second thread for parallel processing; controlling the first thread to run data conversion processing of the first target node; and controlling the second thread to run data conversion processing of the third target node.

[0097] In one embodiment, when the processor executes the computer program, the following steps are also implemented: caching the data of the first object to be converted obtained after the data conversion processing of the first target node; caching the data of the second object to be converted obtained after the data conversion processing of the third target node.

[0098] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining the version of the target model file; recording the version of the target model file in a version information table of the database, wherein the version information table records the versions of one or more model files, and the differences between the versions of the model files include the updating, addition or deletion of some data; reading the corresponding template file based on any version in the version information record table, parsing each data object and the data conversion instructions corresponding to each data object and the metadata of the initial data object from the corresponding template file through a model parser to perform data conversion processing.

[0099] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: determining a data model, the data model is used to describe the data structure of each data object and the relationship between each data object; identifying the file format type of the data file to be converted, and dynamically loading the file parsing component based on the file format type; parsing the data file to be converted by the file parsing component, obtaining the metadata of the initial data object of the data model from the parsing result, configuring the metadata of the initial data object into the data model, and configuring each data conversion instruction of the data model based on the relationship between each data object, and obtaining a target model file after configuration; parsing each data object and the data conversion instruction corresponding to each data object and the metadata of the initial data object from the target model file by a model parser, and constructing an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node; traversing the abstract syntax tree to obtain multiple target nodes and edges connecting each target node, and performing data conversion processing based on the multiple target nodes and the edges connecting each target node and the metadata of the initial data object.

[0100] In one embodiment, when a computer program is executed by a processor to implement the step of dynamically loading a target file parsing component based on a file format type, the following steps are implemented: if the file format type is a JSON type, the loaded target file parsing component is a JACKSON component; if the file format type is a YAML type, the loaded target file parsing component is a SnakeYAML component; if the file format type is an XML type, the loaded target file parsing component is a JAXP component.

[0101] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining the data structure of each node of the abstract syntax tree, and determining the node type of each node based on the data structure of each node; constructing a hash table, and creating each index of the hash table based on each node type; when the processor executes the computer program to implement the step of traversing the abstract syntax tree to obtain multiple target nodes and edges connected to each target node, the following steps are implemented: traversing the abstract syntax tree based on each index of the hash table to obtain multiple target nodes and edges connected to each target node.

[0102] In one embodiment, when a computer program is executed by a processor to implement the step of performing data conversion processing based on multiple target nodes and the edges connecting each target node and the metadata of the initial data object, the following steps are implemented: determining a first target node and multiple second target nodes among the multiple target nodes, the first target node being a node of the first object to be converted, and the multiple second target nodes including a node of the initial data object and each conversion node before the first target node; performing data conversion processing based on the edges connecting each target node and the metadata of the initial data object to obtain conversion data of the object to be converted of the first target node.

[0103] In one embodiment, when a computer program is executed by a processor to implement the step of performing data conversion processing based on multiple target nodes and the edges connecting each target node and the metadata of the initial data object, the computer program also implements the following steps: determining a third target node and multiple fourth target nodes among the multiple target nodes, the third target node being a node of the second object to be converted, and the multiple fourth target nodes including a node of the initial data object and each conversion node before the third target node; starting a first thread and a second thread for parallel processing; controlling the first thread to run data conversion processing of the first target node; and controlling the second thread to run data conversion processing of the third target node.

[0104] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: caching the data of the first object to be converted obtained after the data conversion processing of the first target node; caching the data of the second object to be converted obtained after the data conversion processing of the third target node.

[0105] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining the version of the target model file; recording the version of the target model file in a version information table of the database, wherein the version information table records the versions of one or more model files, and the differences between the versions of the model files include the updating, addition or deletion of some data; reading the corresponding template file based on any version in the version information record table, parsing each data object and the data conversion instructions corresponding to each data object and the metadata of the initial data object from the corresponding template file through a model parser to perform data conversion processing.

[0106] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0107] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A data conversion method for a system data application layer, the method include: Determine a data model, wherein the data model is used to describe the data structure of each data object and the relationship between each data object; Identify the file format type of the data file to be converted, and dynamically load the file parsing component based on the file format type; Parsing the data file to be converted by the file parsing component, obtaining metadata of the initial data object of the data model from the parsing result, configuring the metadata of the initial data object into the data model, and configuring each data conversion instruction of the data model based on the relationship between each data object, and obtaining the target model file after configuration; Parsing each data object and the data conversion instruction corresponding to each data object and metadata of the initial data object from the target model file by a model parser, and constructing an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node; The abstract syntax tree is traversed to obtain a plurality of target nodes and edges connecting the target nodes, and data conversion processing is performed based on the plurality of target nodes, the edges connecting the target nodes, and metadata of the initial data object.

2. The method according to claim 1, It is characterized in that The dynamically loading target file parsing component based on the file format type includes: If the file format type is JSON type, the loaded target file parsing component is the JACKSON component; If the file format type is YAML type, the loaded target file parsing component is the SnakeYAML component; If the file format type is XML type, the loaded target file parsing component is a JAXP component.

3. The method according to claim 1, It is characterized in that After the step of constructing an abstract syntax tree using each data object as a node and each data conversion instruction as an edge connecting each node, the method further includes: Acquire the data structure of each node of the abstract syntax tree, and determine the node type of each node based on the data structure of each node; Constructing a hash table, and creating indexes of the hash table based on the node types; The traversing the abstract syntax tree to obtain a plurality of target nodes and edges connected to the target nodes includes: The abstract syntax tree is traversed based on each index of the hash table to obtain a plurality of target nodes and edges connecting each target node.

4. The method according to claim 1, It is characterized in that The performing of data conversion processing based on the multiple target nodes, the edges connecting the target nodes, and the metadata of the initial data object includes: Determine a first target node and a plurality of second target nodes among the plurality of target nodes, wherein the first target node is a node of a first object to be converted, and the plurality of second target nodes include a node of an initial data object and each conversion node before the first target node; Data conversion processing is performed based on the edges connecting the target nodes and the metadata of the initial data object to obtain conversion data of the object to be converted of the first target node.

5. The method according to claim 4, It is characterized in that The performing of data conversion processing based on the multiple target nodes and the edges connecting the target nodes and the metadata of the initial data object also includes: Determine a third target node and a plurality of fourth target nodes among the plurality of target nodes, wherein the third target node is a node of the second object to be converted, and the plurality of fourth target nodes include a node of the initial data object and each conversion node before the third target node; Starting a first thread and a second thread for parallel processing; Controlling the first thread to execute data conversion processing of the first target node; Control the second thread to execute data conversion processing of the third target node.

6. The method according to claim 5, It is characterized in that The method further comprises: Cache the data of the first object to be converted obtained after the data conversion processing of the first target node; The data of the second object to be converted obtained after the data conversion processing of the third target node is cached.

7. The method according to claim 1, It is characterized in that After obtaining the target model file, the method further includes: Determine the version of the target model file; Recording the version of the target model file in a version information table of the database, wherein the version information table records one or more versions of the model file, and the differences between the versions of the model file include the update, addition or deletion of some data; Based on any version in the version information record table, the corresponding template file is read, and each data object and the data conversion instructions corresponding to each data object and the metadata of the initial data object are parsed from the corresponding template file by a model parser to perform data conversion processing.

8. A data conversion device for a system data application layer, the device include: A determination module, used to determine a data model, wherein the data model is used to describe the data structure of each data object and the relationship between each data object; A loading module, used to identify the file format type of the data file to be converted, and dynamically load the file parsing component based on the file format type; A parsing and configuration module, used to parse the data file to be converted through the file parsing component, obtain metadata of the initial data object of the data model from the parsing result, configure the metadata of the initial data object into the data model, and configure each data conversion instruction of the data model based on the relationship between each data object, and obtain the target model file after configuration; A construction module, used to parse each data object and the data conversion instruction corresponding to each data object and the metadata of the initial data object from the target model file through a model parser, and construct an abstract syntax tree with each data object as a node and each data conversion instruction as an edge connecting each node; The data conversion module is used to traverse the abstract syntax tree to obtain multiple target nodes and edges connecting the target nodes, and perform data conversion processing based on the multiple target nodes and edges connecting the target nodes and metadata of the initial data object.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images